| Summary: | ASTERISK-30343: res_crypto: ast_sign_bin fails with default RedHat crypto policies | ||||
| Reporter: | Michael Newton (miken32) | Labels: | |||
| Date Opened: | 2022-12-07 15:04:15.000-0600 | Date Closed: | |||
| Priority: | Minor | Regression? | Yes | ||
| Status: | Open/New | Components: | Resources/res_crypto | ||
| Versions: | 16.29.0 18.15.1 20.0.1 | Frequency of Occurrence | Constant | ||
| Related Issues: |
| ||||
| Environment: | Alma Linux 9.1 | Attachments: | |||
| Description: | After upgrade from 16.28 to 16.29 our dundi queries stopped working with this error output:
{noformat} WARNING[100840]: res_crypto.c:384 ast_sign_bin: RSA Signature (key gateway) failed -1 NOTICE[100840]: pbx_dundi.c:1366 update_key: Failed to sign key (-1)! NOTICE[100840]: pbx_dundi.c:3376 dundi_send: Failed to send packet to '00:50:56:ae:13:23' {noformat} This appears to be a regression resulting from changes in ASTERISK-30046. | ||||
| Comments: | By: Asterisk Team (asteriskteam) 2022-12-07 15:04:18.049-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Joshua C. Colp (jcolp) 2022-12-07 15:07:33.041-0600 Asterisk 16 does not receive bug fixes. It is now security fix only. Any fix for this would only occur in 18, 20, and master as of this time. By: Michael Newton (miken32) 2022-12-07 15:24:46.255-0600 Hmm, the rather big update of res_crypto didn't seem like a security fix. I guess I'll have to see if the issue can be reproduced in Asterisk 18. By: Joshua C. Colp (jcolp) 2022-12-07 15:28:35.643-0600 Asterisk 16 went security fix October 9th[1]. The change in question occurred before that date. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions By: Michael Newton (miken32) 2022-12-07 15:36:49.466-0600 A reasonable approach might be to support regressions caused by changes made before that cutoff. (Not trying to be difficult, and I know I'm the only person in the world still using dundi lol) By: Joshua C. Colp (jcolp) 2022-12-07 17:00:53.778-0600 That's not something I'm willing to do at this point in time. It becomes: How far back do we then support? What constitutes an actual regression in that case? Do we continue to do releases? How long do we wait for someone to report something? There's a lot involved in it and it's not something we can take on. We already do release candidates so people can test for these kinds of things before releases are done, to try to minimize it so we can fix things before full release. When this is resolved then the patch itself can likely be applied to 16. By: Joshua C. Colp (jcolp) 2022-12-07 17:07:10.693-0600 I will say this, we have not yet done the final release of both 16 and 19 for bug fix releases. If it is resolved before that time, then it will be considered for inclusion. By: Michael Newton (miken32) 2022-12-08 13:02:16.152-0600 Understood, you have to draw the line somewhere. I would like to see support timelines kept in mind when committing potentially breaking changes like this to a branch that's soon to lose support. The rewrite was certainly needed for branches that are going forward, but 16 could have continued just fine with OpenSSL 3.0 and the deprecated functions. I've tried to quickly reproduce in 18 but it doesn't even want to load my keys at all, for whatever reason. I will have time to dig into it in a few weeks if it's still needed. I'm not a C programmer, but I do know how to fumble my way around gdb, and can hopefully find where this patch broke things. We package our own RPMs so can backport it, even if it doesn't make its way into a release. By: Joshua C. Colp (jcolp) 2022-12-08 13:25:19.048-0600 Yes, this will be kept in mind for the future and the policy may be adjusted/extended. By: Philip Prindeville (pprindeville) 2022-12-11 23:36:45.040-0600 @miken32 Can you build and run a debug version, and breakpoint it? I don't have dundi configured or anyone to peer with. But some breakpoints and value snapshots would help... By: Michael Newton (miken32) 2022-12-12 18:39:40.444-0600 Yes I will get that set up within the next couple of weeks and report back. Will values passed to ast_sign_bin (res_crypto.c:354) and evp_pkey_sign (res_crypto.c:315) be sufficient? Let me know if anything else is needed. By: George Joseph (gjoseph) 2022-12-13 09:48:14.302-0600 I can't seem to reproduce this between Fedora 37 (openssl 3.0.5 5) and CentOS 7 (openssl 1.0.2k-fips) on any version of Asterisk. Can you provide a sample dundi.conf and a set of SAMPLE keys that reproduces the issue? What version of openssl are you running? Can you give me the exact commands you used to generate your keys? Oh, does a simple {{dundi lookup 123456}} from the CLI cause the issue? By: Michael Newton (miken32) 2022-12-13 11:57:45.029-0600 Any dundi commands that involve talking with another peer trigger the issue. I'm on Alma Linux 9, currently OpenSSL 3.0.1.43. The keys were generated years ago, but I expect they were done with the standard astkeygen script. Here's the key {noformat} -----BEGIN RSA PRIVATE KEY----- MIICXQIBAAKBgQC/EIbx93PN/VbNY98Kbg8wEmkB9pjPAZFognaSK23bO30i8cme R29U58oqi1Dn+MOfvHzWUo2uez4rAVbO0lUDKSDZZIzYPVwbzMkNtyy2Ej36ySb0 m/WGvMCpEprbUYs4jY/Q1KM5o74Qv2Aywy032Q3UWh9cMmsdWZ046ndPBwIDAQAB AoGAXaRD/yNAZpzbhh6EmiAG4ZCkVon9qrciBQ6r/ke6t9AYLKBEKIbqUbqoouFU 7dxGRGuk44XiWrmcZodpfEQp1WGIdtrWlZ7NOIp5kA5FO9uUcX5hfCxZH9h68j1L cdNgb3tZDLjBMT6oipxEyEOcxjvkIaIL11zyx0SqCndoxUECQQDrK39wiUnYjDQW 1fEPjrRX/zk7kOWKa5Zt6fj5BqctmHHhDD7hyFAwVwcOeSmQZm2yPoakLXBrQPCi v0kdoEvDAkEAz/zvktzZhmQ33FEjmyRFLhimfvYP+nqaSB8ZPV4DkGM6PgMXYsWp 4eLJowHoKsQpJmiKbPV6+on5Q4NW49TvbQJAIBGxcj42hMIxxD9ufQmfzDQwsM/E jYi4Xcq/Oe5PU+dq+B58YLu5O65SdwXMxjVBlkHyiGbt4qJbbkYZiWG3kwJBAJnA Ul4P0uHtLfo5JQgn7NghstsCDVfN0EVmb+MUn6/aGpEC+gOzOV1ZqFNPMpCCyCSz fTkE0y9oVZLaAZ6Up5UCQQCwFq9xZ+Co0xCN4r8hvY3DCUIYK8GwyqJZno4w975E tiWZkGKsuJpHPUJAGzs/t9s7h73/oXybE77YLtqHTy3/ -----END RSA PRIVATE KEY----- -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC/EIbx93PN/VbNY98Kbg8wEmkB 9pjPAZFognaSK23bO30i8cmeR29U58oqi1Dn+MOfvHzWUo2uez4rAVbO0lUDKSDZ ZIzYPVwbzMkNtyy2Ej36ySb0m/WGvMCpEprbUYs4jY/Q1KM5o74Qv2Aywy032Q3U Wh9cMmsdWZ046ndPBwIDAQAB -----END PUBLIC KEY----- {noformat} And the config. Pretty basic; we've patched dundi in a very rudimentary way so it works with pjsip, but that didn't involve any changes to the crypto code at all. {noformat} [general] organization=Example Inc. locality=Toronto stateprov=ON country=CA email=info@example.com phone=+18445551212 bindaddr = 10.10.241.120 port = 4520 tos = ef cachetime = 1800 ttl = 1 autokill = yes secretpath = gateway storehistory = yes [mappings] ; how we answer queries from the PBXs ; context => ast context,weight,tech,destination,option,option... outbound => outbound-dundi-lookup,5,PJSIP,${NUMBER}@Gateway_1,nopartial [dundi-default](!) ; template for PBX entries qualify=yes model=symmetric order=primary inkey=pbx outkey=gateway permit=all include=all [00:50:56:ae:26:74](dundi-default) host=10.10.241.21 {noformat} By: George Joseph (gjoseph) 2022-12-13 12:40:16.142-0600 Still can't reproduce with the key and config you gave me. Can you revert your patches please and try again. Also what are the permissions on the key files? and...where does astkeydir in asterisk.conf point to and what are its permissions? By: Michael Newton (miken32) 2022-12-13 14:37:32.992-0600 No permissions problems, and SELinux is not interfering either; same problem with it set to permissive. {noformat} $ ls -lahZ /var/lib/asterisk/keys/ total 12K drwxr-x---. 3 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 78 Sep 15 17:06 . drwxr-x---. 12 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 160 Sep 15 17:06 .. -r--------. 1 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 887 Sep 8 12:19 gateway.key -r--r--r--. 1 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 271 Sep 24 2015 gateway.pub -r--r--r--. 1 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 271 Sep 24 2015 pbx.pub drwxr-x---. 2 asterisk asterisk system_u:object_r:asterisk_var_lib_t:s0 6 Sep 15 17:06 stir_shaken {noformat} By: Michael Newton (miken32) 2022-12-13 15:04:47.358-0600 I'll get a test environment set up in the next 10-15 days. The patch we've applied is this one: https://gerrit.asterisk.org/c/asterisk/+/10855/ By: Michael Newton (miken32) 2023-03-20 13:37:12.318-0500 TLDR: {{update-crypto-policies --set LEGACY}} Rather delayed, but I got a test install set up, using Asterisk 16.30.0. After stepping through the Asterisk code I got as far as {{res_crypto.c:360}} calling OpenSSL function {{EVP_PKEY_CTX_set_signature_md()}} which was failing. {noformat} Thread 27 "asterisk" hit Breakpoint 1, evp_pkey_sign (pkey=0x5555564fb100, in=0x7ffff60fdef0 "\r8\274AW\027,W\374I\a\246\232~\242F\343\274<\210\200", inlen=20, sig=0x555555e4573c "", siglen=0x7ffff60fdedc, padding=1) at /usr/src/debug/asterisk-16.30.0-0.el9.x86_64/res/res_crypto.c:360 360 /usr/src/debug/asterisk-16.30.0-0.el9.x86_64/res/res_crypto.c: No such file or directory. (gdb) stepi 0x00007ffff64cf650 in EVP_sha1@plt () from /usr/lib64/asterisk/modules/res_crypto.so (gdb) next Single stepping until exit from function EVP_sha1@plt, which has no line number information. EVP_sha1 () at crypto/evp/legacy_sha.c:98 98 { {noformat} Despite having all the debuginfo packages installed I couldn't get in there, but seeing {{legacy_sha.c}} reminded me of trouble we had with using older RSA keys with SHA1 signatures for SSH on one of our servers (and the temporary solution.) So in the end, I was able to resolve the issue by running {{update-crypto-policies --set LEGACY}}. The man pages have this to say about OpenSSL in particular: {noformat} * Applications using OpenSSL: If an application allows the configuration of ciphersuite string, the special cipher string "PROFILE=SYSTEM" should replace any other cipher string. Applications which use the default library settings automatically adhere to the policy. Applications following the policy inherit the settings for cipher suite preference. By default the OpenSSL library reads a configuration file when it is initialized. If the application does not override loading of the configuration file, the policy also sets the minimum TLS protocol version and default cipher suite preference via this file. If the application is long-running such as the httpd server it has to be restarted to reload the configuration file after policy is changed. Otherwise the changed policy cannot take effect. {noformat} Globally downgrading the system security is not an ideal solution; I guess the proper fix is to improve the security of Asterisk's keys, but in the short term I think this can be addressed with documentation updates. | ||||