Re: [squid-users] problems with ssl_crtd

On 19/09/12 16:46, Guy Helmer wrote:
>
> On Sep 19, 2012, at 9:03 AM, Linos <info_at_linos.es> wrote:
>
>> On 19/09/12 15:30, Guy Helmer wrote:
>>> On Sep 19, 2012, at 5:44 AM, Linos <info_at_linos.es> wrote:
>>>
>>>> Hi,
>>>> i have been using Squid squid-3.2.0.17-20120527-r11561 in an Ubuntu Server
>>>> 12.04 some time with ssl-bump without problems for a year, the ca cert expired
>>>> some days ago and with the new ca cert i installed squid 3.2.1 stable.
>>>>
>>>> Now the proxy exists every time 10 or more users use https at the same time,
>>>> it's pretty strange, i have tried to downgrade to the old squid version but i
>>>> can't get the proxy to be stable no matter if using new or old version, i have
>>>> tried to recreate other cert just in case, same problem, i recreated too
>>>> squid_ssl_db and cache_dir, no matter what i do it keeps crashing, the cache log
>>>> read as this:
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> 2012/09/19 11:58:00| Starting Squid Cache version 3.2.1 for x86_64-pc-linux-gnu...
>>>> 2012/09/19 11:58:00| Process ID 30077
>>>> 2012/09/19 11:58:00| Process Roles: master worker
>>>> 2012/09/19 11:58:00| With 65535 file descriptors available
>>>> 2012/09/19 11:58:00| Initializing IP Cache...
>>>> 2012/09/19 11:58:00| DNS Socket created at [::], FD 4
>>>> 2012/09/19 11:58:00| DNS Socket created at 0.0.0.0, FD 5
>>>> 2012/09/19 11:58:00| Adding nameserver 80.58.61.250 from squid.conf
>>>> 2012/09/19 11:58:00| Adding nameserver 8.8.8.8 from squid.conf
>>>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/10 'ssl_crtd' processes
>>>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/20 'request_body_max_size.sh'
>>>> processes
>>>> 2012/09/19 11:58:00| Logfile: opening log daemon:/var/log/squid3/access.log
>>>> 2012/09/19 11:58:00| Logfile Daemon: opening log /var/log/squid3/access.log
>>>> 2012/09/19 11:58:00| Unlinkd pipe opened on FD 31
>>>> 2012/09/19 11:58:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
>>>> 2012/09/19 11:58:00| Store logging disabled
>>>> 2012/09/19 11:58:00| Swap maxSize 15360000 + 262144 KB, estimated 312442 objects
>>>> 2012/09/19 11:58:00| Target number of buckets: 15622
>>>> 2012/09/19 11:58:00| Using 16384 Store buckets
>>>> 2012/09/19 11:58:00| Max Mem size: 262144 KB
>>>> 2012/09/19 11:58:00| Max Swap size: 15360000 KB
>>>> 2012/09/19 11:58:00| Rebuilding storage in /mnt/squid/squid3 (clean log)
>>>> 2012/09/19 11:58:00| Using Least Load store dir selection
>>>> 2012/09/19 11:58:00| Set Current Directory to /mnt/squid/squid3
>>>> 2012/09/19 11:58:00| Loaded Icons.
>>>> 2012/09/19 11:58:00| HTCP Disabled.
>>>> 2012/09/19 11:58:00| Squid plugin modules loaded: 0
>>>> 2012/09/19 11:58:00| Adaptation support is off.
>>>> 2012/09/19 11:58:00| Accepting NAT intercepted HTTP Socket connections at
>>>> local=0.0.0.0:3128 remote=[::] FD 36 flags=41
>>>> 2012/09/19 11:58:00| Accepting SSL bumped HTTP Socket connections at
>>>> local=[::]:3150 remote=[::] FD 37 flags=9
>>>> 2012/09/19 11:58:00| Store rebuilding is 16.55% complete
>>>> 2012/09/19 11:58:00| Done reading /mnt/squid/squid3 swaplog (24167 entries)
>>>> 2012/09/19 11:58:00| Finished rebuilding storage from disk.
>>>> 2012/09/19 11:58:00| 24167 Entries scanned
>>>> 2012/09/19 11:58:00| 0 Invalid entries.
>>>> 2012/09/19 11:58:00| 0 With invalid flags.
>>>> 2012/09/19 11:58:00| 24167 Objects loaded.
>>>> 2012/09/19 11:58:00| 0 Objects expired.
>>>> 2012/09/19 11:58:00| 0 Objects cancelled.
>>>> 2012/09/19 11:58:00| 0 Duplicate URLs purged.
>>>> 2012/09/19 11:58:00| 0 Swapfile clashes avoided.
>>>> 2012/09/19 11:58:00| Took 0.12 seconds (204025.29 objects/sec).
>>>> 2012/09/19 11:58:00| Beginning Validation Procedure
>>>> 2012/09/19 11:58:00| Completed Validation Procedure
>>>> 2012/09/19 11:58:00| Validated 24167 Entries
>>>> 2012/09/19 11:58:00| store_swap_size = 732468.00 KB
>>>> 2012/09/19 11:58:01| storeLateRelease: released 0 objects
>>>> (ssl_crtd): Cannot create ssl certificate or private key.
>>>> 2012/09/19 12:03:20| WARNING: ssl_crtd #1 exited
>>>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>>>> 2012/09/19 12:03:20| Starting new helpers
>>>> 2012/09/19 12:03:20| helperOpenServers: Starting 1/10 'ssl_crtd' processes
>>>> 2012/09/19 12:03:20| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper
>>>> return <NULL> reply
>>>> (ssl_crtd): Cannot create ssl certificate or private key.
>>>> 2012/09/19 12:03:20| WARNING: ssl_crtd #2 exited
>>>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>>>> 2012/09/19 12:03:20| Closing HTTP port 0.0.0.0:3128
>>>> 2012/09/19 12:03:20| Closing HTTP port [::]:3150
>>>> 2012/09/19 12:03:20| storeDirWriteCleanLogs: Starting...
>>>> 2012/09/19 12:03:20| Finished. Wrote 24195 entries.
>>>> 2012/09/19 12:03:20| Took 0.02 seconds (1321120.45 entries/sec).
>>>> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
>>>>
>>>> Squid Cache (Version 3.2.1): Terminated abnormally.
>>>> CPU Usage: 1.896 seconds = 0.740 user + 1.156 sys
>>>> Maximum Resident Size: 144640 KB
>>>> Page faults with physical i/o: 0
>>>> Memory usage for squid via mallinfo():
>>>> total space in arena: 18900 KB
>>>> Ordinary blocks: 18674 KB 54 blks
>>>> Small blocks: 0 KB 1 blks
>>>> Holding blocks: 37552 KB 9 blks
>>>> Free Small blocks: 0 KB
>>>> Free Ordinary blocks: 225 KB
>>>> Total in use: 56226 KB 297%
>>>> Total free: 225 KB 1%
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> configure and kernel versions:
>>>>
>>>> kernel: 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64
>>>> x86_64 x86_64 GNU/Linux
>>>>
>>>> Squid Cache: Version 3.2.1
>>>> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
>>>> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
>>>> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
>>>> '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode'
>>>> '--disable-dependency-tracking' '--disable-silent-rules'
>>>> '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
>>>> '--mandir=/usr/share/man' '--with-cppunit-basedir=/usr' '--enable-inline'
>>>> '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd'
>>>> '--enable-removal-policies=lru,heap' '--enable-delay-pools'
>>>> '--enable-cache-digests' '--enable-underscores' '--enable-icap-client'
>>>> '--enable-follow-x-forwarded-for' '--enable-auth-basic' '--enable-auth-digest'
>>>> '--enable-auth-ntlm' '--enable-auth-negotiate'
>>>> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
>>>> '--enable-ntlm-auth-helpers=smb_lm,'
>>>> '--enable-digest-auth-helpers=ldap,password'
>>>> '--enable-negotiate-auth-helpers=squid_kerb_auth'
>>>> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
>>>> '--enable-arp-acl' '--enable-esi' '--enable-zph-qos' '--enable-wccpv2'
>>>> '--disable-translation' '--with-logdir=/var/log/squid3'
>>>> '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536'
>>>> '--with-large-files' '--with-default-user=proxy' '--enable-ssl'
>>>> '--enable-ssl-crtd' '--disable-epoll' '--enable-linux-netfilter'
>>>> 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector
>>>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>>>> 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now'
>>>> 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector
>>>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> i get this in dmesg:
>>>>
>>>> [3433312.743391] init: squid3 main process (29801) terminated with status 1
>>>> [3433417.181960] ssl_crtd[29848]: segfault at 0 ip 00007f2ee2494ac5 sp
>>>> 00007fff51dbd260 error 4 in ssl_crtd[7f2ee248b000+10000]
>>>> [3433417.328898] init: squid3 main process (29847) terminated with status 1
>>>> [3433557.765878] init: squid3 main process (29877) killed by KILL signal
>>>> [3433944.030777] init: squid3 main process (30077) terminated with status 1
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>
>>> OK, ssl_crtd is dying with a segfault when it is handling a request.
>>>
>>> What is in the squid_ssl_db/size file? If it is empty (0 bytes), that would cause a crash in older (pre-3.2.1) squid.
>>>
>>> Otherwise, is there an ssl_crtd core file in the squid log directory? A stack trace from it might help diagnose the problem.
>>>
>>> Guy
>>>
>>
>> Thanks for reply.
>>
>> i checked the squid_ssl_db/size because i found the empty file problem searching
>> for my own problem in the mailing list, it's ok in my host, the file have the
>> content "139264" right now.
>>
>> I can't found the core file, do i need to do something for it to generate? maybe
>> a configure script option or squid.conf change to activate it?
>>
>> Regards,
>> Miguel Angel.
>
> I have
>
> coredump_dir /var/log/squid
>
> to get coredumps in my /var/log/squid directory. Now that I think about it, I don't remember if this works for ssl_crtd though -- seems like I have had to start "gdb ssl_crtd" and then attach to one of the ssl_crtd processes, then generate HTTPS traffic to trigger the request to ssl_crtd and get a backtrace when ssl_crtd gets the segfault signal�
>
> Guy
>

I have defined a coredump_dir but don't seems to work, probably squid would only
coredump itself, i will try to obtain a backtrace with gdb.

Miguel Angel.
Received on Wed Sep 19 2012 - 15:42:28 MDT