Re: [squid-users] problems with ssl_crtd

On 24/09/2012 8:44 p.m., Linos wrote:
> On 20/09/12 12:58, Ahmed Talha Khan wrote:
>> Hey Guy, All
>>
>> I have started facing a very similar issue now.I have been using
>> squid-3.HEAD-20120421-r12120 for about 5 months without any issues.
>> Suddenly from yesterday ive started getting crahses in ssl_crtd
>> process.
>>
>>
>> In my case i am the only user but i observe that the behaviour is
>> random. Sometimes it crashes and sometimes it works. Different https
>> pages give the crash. Even non https pages have caused the crash.
>>
>> These occur especially on google https pages like docs,mail,calender etc..
>>
>> The signing cert is also ok and has NOT expired.
>>
>>
>> My squid conf looks like this:
>> *******************************************************
>> sslproxy_cert_error allow all
>>
>> sslcrtd_program /usr/local/squid-3.3/libexec/ssl_crtd -s
>> /usr/local/squid-3.3/var/lib/ssl_db -M 4MB
>> sslcrtd_children 5
>>
>> http_port 192.168.8.134:3128 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=4MB
>> cert=/home/asif/squid/www.sample.com.pem
>> key=/home/asif/squid/www.sample.com.pem
>>
>> http_port 192.168.8.134:8080
>>
>> https_port 192.168.8.134:3129 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=4MB
>> cert=/home/asif/squid/www.sample.com.pem
>> key=/home/asif/squid/www.sample.com.pem
>> *******************************************************
>>
>> The ssl_db directory is initialized properly with correct permissions.
>>
>> ***********************************************************
>> [talha_at_localhost lib]$ pwd
>> /usr/local/squid-3.3/var/lib
>>
>> [talha_at_localhost lib]$ ls -al
>> total 24
>> drwxrwxrwx 3 root root 4096 Sep 20 15:31 .
>> drwxrwxrwx 6 root root 4096 Sep 20 15:05 ..
>> drwxrwxrwx 3 nobody talha 4096 Sep 20 15:31 ssl_db
>>
>> The size file also has some values in it and cert generation also
>> seems to work but suddenly it all crashes .
>> **************************************************************
>>
>>
>>
>> 2012/09/20 14:57:45| Starting Squid Cache version
>> 3.HEAD-20120425-r12120 for x86_64-unknown-linux-gnu...
>> 2012/09/20 14:57:45| Process ID 23826
>> 2012/09/20 14:57:45| Process Roles: master worker
>> 2012/09/20 14:57:45| With 1024 file descriptors available
>> 2012/09/20 14:57:45| Initializing IP Cache...
>> 2012/09/20 14:57:45| DNS Socket created at [::], FD 5
>> 2012/09/20 14:57:45| DNS Socket created at 0.0.0.0, FD 6
>> 2012/09/20 14:57:45| Adding nameserver 192.168.8.1 from /etc/resolv.conf
>> 2012/09/20 14:57:45| Adding domain localdomain from /etc/resolv.conf
>> 2012/09/20 14:57:45| helperOpenServers: Starting 5/5 'ssl_crtd' processes
>> 2012/09/20 14:57:45| Logfile: opening log
>> daemon:/usr/local/squid-3.3/var/logs/access.log
>> 2012/09/20 14:57:45| Logfile Daemon: opening log
>> /usr/local/squid-3.3/var/logs/access.log
>> 2012/09/20 14:57:45| Logfile: opening log /usr/local/squid-3.3/var/logs/icap-log
>> 2012/09/20 14:57:45| WARNING: log parameters now start with a module
>> name. Use 'stdio:/usr/local/squid-3.3/var/logs/icap-log'
>>
>>
>> 2012/09/20 14:57:45| Store logging disabled
>> 2012/09/20 14:57:45| Swap maxSize 0 + 262144 KB, estimated 20164 objects
>> 2012/09/20 14:57:45| Target number of buckets: 1008
>> 2012/09/20 14:57:45| Using 8192 Store buckets
>> 2012/09/20 14:57:45| Max Mem size: 262144 KB
>> 2012/09/20 14:57:45| Max Swap size: 0 KB
>> 2012/09/20 14:57:45| Using Least Load store dir selection
>> 2012/09/20 14:57:45| Set Current Directory to /usr/local/squid-3.3/var/cache
>> 2012/09/20 14:57:45| Loaded Icons.
>> 2012/09/20 14:57:45| HTCP Disabled.
>> 2012/09/20 14:57:45| /usr/local/squid-3.3/var/run/squid.pid: (13)
>> Permission denied
>> 2012/09/20 14:57:45| WARNING: Could not write pid file
>> 2012/09/20 14:57:45| Squid plugin modules loaded: 0
>> 2012/09/20 14:57:45| Adaptation support is on
>> 2012/09/20 14:57:45| Accepting SSL bumped HTTP Socket connections at
>> local=192.168.8.134:3128 remote=[::] FD 20 flags=9
>> 2012/09/20 14:57:45| Accepting HTTP Socket connections at
>> local=192.168.8.134:8080 remote=[::] FD 21 flags=9
>> 2012/09/20 14:57:45| Accepting SSL bumped HTTPS Socket connections at
>> local=192.168.8.134:3129 remote=[::] FD 22 flags=9
>> 2012/09/20 14:57:46| storeLateRelease: released 0 objects
>>
>> (ssl_crtd): Cannot create ssl certificate or private key.
>> 2012/09/20 14:58:23| WARNING: ssl_crtd #2 exited
>> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5)
>>
>> 2012/09/20 14:58:23| Starting new helpers
>> 2012/09/20 14:58:23| helperOpenServers: Starting 1/5 'ssl_crtd' processes
>> 2012/09/20 14:58:23| client_side.cc(3478) sslCrtdHandleReply:
>> "ssl_crtd" helper return <NULL> reply
>> (ssl_crtd): Cannot create ssl certificate or private key.
>>
>> 2012/09/20 14:58:23| WARNING: ssl_crtd #1 exited
>> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5)
>> 2012/09/20 14:58:23| storeDirWriteCleanLogs: Starting...
>> 2012/09/20 14:58:23| Finished. Wrote 0 entries.
>> 2012/09/20 14:58:23| Took 0.00 seconds ( 0.00 entries/sec).
>> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
>>
>> Squid Cache (Version 3.HEAD-20120425-r12120): Terminated abnormally.
>> CPU Usage: 0.355 seconds = 0.289 user + 0.066 sys
>> Maximum Resident Size: 71104 KB
>> Page faults with physical i/o: 0
>> Memory usage for squid via mallinfo():
>> total space in arena: 11924 KB
>> Ordinary blocks: 11818 KB 49 blks
>> Small blocks: 0 KB 0 blks
>> Holding blocks: 664 KB 2 blks
>> Free Small blocks: 0 KB
>> Free Ordinary blocks: 105 KB
>>
>>
>>
>>
>>
>>
>> On Thu, Sep 20, 2012 at 2:52 PM, Linos wrote:
>>> On 19/09/12 16:46, Guy Helmer wrote:
>>>>> Thanks for reply.
>>>>>
>>>>> i checked the squid_ssl_db/size because i found the empty file problem searching
>>>>> for my own problem in the mailing list, it's ok in my host, the file have the
>>>>> content "139264" right now.
>>>>>
>>>>> I can't found the core file, do i need to do something for it to generate? maybe
>>>>> a configure script option or squid.conf change to activate it?
>>>>>
>>>>> Regards,
>>>>> Miguel Angel.
>>>> I have
>>>>
>>>> coredump_dir /var/log/squid
>>>>
>>>> to get coredumps in my /var/log/squid directory. Now that I think about it, I don't remember if this works for ssl_crtd though -- seems like I have had to start "gdb ssl_crtd" and then attach to one of the ssl_crtd processes, then generate HTTPS traffic to trigger the request to ssl_crtd and get a backtrace when ssl_crtd gets the segfault signal�
>>>>
>>>> Guy
>>>>
>>> Hi,
>>> i have been trying to debug with gdb attaching existing process, the strange
>>> it's that ssl_ctrd seems to exit normally in this test, here you have it (sorry
>>> for the spanish locale, i will use english next time, the only file with symbols
>>> it's ssl_crtd itself):
>>>
>>> --------------------------------------------------------------------------------
>>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> Para las instrucciones de informe de errores, vea:
>>> <http://bugs.launchpad.net/gdb-linaro/>.
>>> (gdb) attach 10495
>>> Adjuntando a process 10495
>>> Leyendo s�mbolos desde /usr/lib/squid3/ssl_crtd...Leyendo s�mbolos desde
>>> /usr/lib/debug/usr/lib/squid3/ssl_crtd...hecho.
>>> hecho.
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libc.so.6
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libz.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libm.so.6
>>> Leyendo s�mbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron s�mbolos
>>> de depuraci�n)hecho.
>>> S�mbolos cargados para /lib64/ld-linux-x86-64.so.2
>>> 0x00007f3ef414f0a0 in read () from /lib/x86_64-linux-gnu/libc.so.6
>>> (gdb) continue
>>> Continuando.
>>> [Inferior 1 (process 10495) exited normally]
>>> (gdb) bt
>>> No stack.
>>>
>>> --------------------------------------------------------------------------------
>>>
>>> I have tried attaching to squid3 process itself and i have received a signal here:
>>>
>>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> Para las instrucciones de informe de errores, vea:
>>> <http://bugs.launchpad.net/gdb-linaro/>.
>>> (gdb) attach 10732
>>> Adjuntando a process 10732
>>> Leyendo s�mbolos desde /usr/sbin/squid3...coLeyendo s�mbolos desde
>>> /usr/lib/debug/usr/sbin/squid3...ntinue
>>> hecho.
>>> hecho.
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libpthread.so.0...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> [Depuraci�n de hilo usando libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libpthread.so.0
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libxml2.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libexpat.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libexpat.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libssl.so.0.9.8...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libssl.so.0.9.8
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5.so.3
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libcom_err.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libcom_err.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libcap.so.2...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libcap.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/librt.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/librt.so.1
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libltdl.so.7...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libltdl.so.7
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libm.so.6
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libc.so.6
>>> Leyendo s�mbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron s�mbolos
>>> de depuraci�n)hecho.
>>> S�mbolos cargados para /lib64/ld-linux-x86-64.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libz.so.1
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
>>> Leyendo s�mbolos desde /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libkeyutils.so.1...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libkeyutils.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libresolv.so.2...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libresolv.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libnss_files.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libnss_files.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libnss_compat.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libnss_compat.so.2
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libnsl.so.1...(no se encontraron
>>> s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libnsl.so.1
>>> Leyendo s�mbolos desde /lib/x86_64-linux-gnu/libnss_nis.so.2...(no se
>>> encontraron s�mbolos de depuraci�n)hecho.
>>> S�mbolos cargados para /lib/x86_64-linux-gnu/libnss_nis.so.2
>>> 0x00007f7d6243dac8 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>>> (gdb) continue
>>> Continuando.
>>>
>>> Program received signal SIGPIPE, Broken pipe.
>>> 0x00007f7d647becb0 in __write_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> (gdb) bt
>>> #0 0x00007f7d647becb0 in __write_nocancel () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> #1 0x00007f7d63d075c5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>>> #2 0x00007f7d63d05247 in BIO_write () from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>>> #3 0x00007f7d63ffafc4 in ssl3_write_pending () from
>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>>> #4 0x00007f7d63ffc853 in ssl3_dispatch_alert () from
>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>>> #5 0x00007f7d63ff9442 in ssl3_shutdown () from
>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>>> #6 0x00007f7d64e8f0f4 in AsyncCall::make (this=0x7f7d687eb390) at AsyncCall.cc:36
>>> #7 0x00007f7d64e92117 in AsyncCallQueue::fireNext (this=<optimized out>) at
>>> AsyncCallQueue.cc:54
>>> #8 0x00007f7d64e92270 in AsyncCallQueue::fire (this=0x7f7d66f5f2c0) at
>>> AsyncCallQueue.cc:40
>>> #9 0x00007f7d64d7c494 in EventLoop::runOnce (this=0x7fff630b3e60) at
>>> EventLoop.cc:131
>>> #10 0x00007f7d64d7c568 in EventLoop::run (this=0x7fff630b3e60) at EventLoop.cc:95
>>> #11 0x00007f7d64ddc039 in SquidMain (argc=<optimized out>, argv=<optimized out>)
>>> at main.cc:1500
>>> #12 0x00007f7d64d10b76 in SquidMainSafe (argv=<optimized out>, argc=<optimized
>>> out>) at main.cc:1215
>>> #13 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1207
>>>
>>> Any ideas what's going on with this information? Thansk!
>>>
>>> Regards,
>>> Miguel Angel.
>>
>>
> Hi Ahmed,
> did you found a way to fix or workaround this? i can't get a backtrace with the
> information needed and no matter what i try it keeps failing, i had to disable
> ssl and my users are a bit upset.
>
> Regards,
> Miguel Angel.

I've taken a quick look at these reports and tried to figure out if
anything obvious is going on.

 From what I can see, there is really no reason why the helper should be
exiting like this. Something it is doing produces an exception, fine,
but the method of handling that exception is to exit instead of
reporting the problem to Squid and continuing with further requests.

As a workaround you can probably patch out the FATAL and let Squid
continuously restart the helpers unless that spams the logs too much for
your liking.

Amos
Received on Mon Sep 24 2012 - 10:52:26 MDT