%include "default.mgp" %default 1 bgrad 0 0 128 90 1 "blue" "grey" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page %nodefault %center, size 7, font "standard", fore "white", vgap 20 Getting Erlang to talk to C and C++ %size 5 from ei to UBF %fore "yellow", size 5 Hal Snyder Vail Systems, Inc. hal@vailsys.com Rick Pettit Vail Systems, Inc. rpettit@vailsys.com %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page what this talk is about current platform at Vail Systems the problem: C/C++ and Erlang first approach: ad hoc second approach: ei, C nodes third approach: UBF conclusions %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Vail Systems computer telephony applications service provider voice over IP (SIP) custom IVR and VoiceXML 2 sites, 3 million calls / day OTP for distributed services: LCR, CDR, RM, stats %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page the problem: C/C++ and Erlang existing systems in separate camps: C/C++ and OTP connecting the two has been expensive in the past new apps: which way to go? unpleasant either/or one goal: C/C++ camp using mnesia with minimal Erlang %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page first approach: ad hoc description reasons results %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ad hoc: description three tiers C++ app OTP request broker same host as C++ app remote OTP node ad hoc protocol - went through two major versions %size 4,prefix " ",font "typewriter" request: reply: %font "standard" adds 1 thread for heartbeat C++, heavy use of STL, fancy OO wrappers (functors) for callbacks %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ad hoc: reasons C++/STL/threads trusted TCP and ad hoc protocols trusted OTP ... risky %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ad hoc: results initial use: least cost routing of outbound calls 5 months to write and test 1274 lines C++ source, plus 1521 lines regression test 1 year in production 48 million requests 0 errors 0 downtime %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page second approach: ei, C nodes description reasons results %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: description two tiers C++ client calling Mod,Func,Arg remote OTP node OTP resources already in use, at known "intentional" DNS names software stack on C++ client C++ app C++ driver with app-specific objects our C driver, generic - heartbeat, reply matchup Ericsson's ei_rpc each application thread calls M,F,A, waits for response %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: reasons greater confidence in OTP incidental: shared lib for ad hoc method developed build problems didn't want to write another app-specific driver ei_rpc uses proven protocol - OTP transport Ericsson wrote a lot of code for us OTP server code - erts C client code - ei no change to existing OTP services %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: results (1 of 3) ERL_TICK => pthreads hell (again) working transactions after two days of coding memory management - terms, messages, queues still finding thread-related problems 4 weeks later message sequencing - feels like rewriting TCP every time we do one of these %center,size 4 mantra of concurrency: no shared data %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: results (2 of 3) message sequencing with ei_rpc %size 4 %font "typewriter" %prefix " " server: %prefix " " add1(Num) -> timer:sleep(2000), Num + 1. %prefix " " client: %prefix " " >./ei_rpc_cli connected to node1@fafner.vail fd=3 0 + 1 = ? ERL_TIMEOUT 1 + 1 = ? ERL_TIMEOUT 2 + 1 = 1 3 + 1 = 2 4 + 1 = ? ERL_TIMEOUT 5 + 1 = 4 6 + 1 = 3 7 + 1 = 5 8 + 1 = ? ERL_TIMEOUT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: results (3 of 3) message sequencing with ei_rpc OTP uses PIDs as transaction IDs C nodes don't have real PIDs work-around: in ei_connect.c, delete %size 5,prefix " ",font "typewriter" self->num = fd; %size 5,prefix " ",font "standard" from ei_reg_send() and ei_rpc_to() %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ei, C nodes: results (concluded) initial use: C++ call control engine OTP resource for call setup info integration in progress all interface code: 5 weeks, so far, still debugging C C shim 2698 lines, plus 1571 lines test C++ app interface 433 lines, plus 77 lines test %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page third approach: UBF description reasons results %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page UBF: description two tiers C++ server doing telephony control OTP client coordinating work requests, resource management C++ server lots of threads for telephony RTP streams etc. one thread iterative TCP UBF server for work requests OTP client web etc. distributed resource for command and status %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page UBF: reasons let us write apps in OTP instead of threaded C++ better security than ei_rpc allow varied endpoints, e.g. Java client, C++ server self-documenting protocol, contract checker %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page UBF: results (1 of 3) initial use: call bridging engine OTP gets work requests from outside, tracks resources C++ does outdial, patches calls together %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page UBF: results (2 of 3) all work so far is on C++ server UBF(B) grammar - 1 hour Erlang simulator plugin - 3 hours flex grammar - 4 hours bison grammar - 8 hours hooks to threaded C++ app - weeks %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page UBF: results (3 of 3) UBF experience so far want to separate contract checker and UBF server backward lists unfriendly to pipelines - [ a b c ] alternative? similarly, should semantic tag precede value it modifies? allow alternative radix on integers - 0x? 16#? some payoff already UBF(B) grammar/contract is checkable/executable Erlang simulator also helps validate design %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page conclusions C++/OTP interface needed, must appeal to non-OTP-zealots ad hoc TCP protocol was very reliable, but great inertia to replicating the approach ei_rpc some implementation surprises, added to toolkit but not dernier cri UBF - very helpful in design and early implementation, ... to be continued ... %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page links %size 4 Joe Armstrong's UBF site: http://www.sics.se/~joe/ubf/site/home.html "Distribution by another means" thread on erlang-questions http://www.erlang.org/ml-archive/erlang-questions/200006/msg00020.html ERLANGS EXTERNAL FORMAT and distribution protocol otp_src_R9C-0/erts/emulator/internal_doc/erl_ext_dist.txt follow-up and code for this presentation will be at http://www.drxyzzy.org/euc2003 %size 4,center Those who ignore Erlang are doomed to repeat it.