The Asterisk Saga

We worked on an Asterisk setup project two weekends ago. It was a rocking experience. And Ameya, one of the heroes in the story, has written up a nice saga on it! Credits to the whole team that worked on it, and thanks to Ameya for writing this up!

Here you go on the Asterisk Saga!

It was Friday evening, Time to party. Not just yet! We came to know about this project. The Client’s requirement was to setup a working Asterisk Voice over IP server. This would enable them to route internal and external calls using a computer, a Digium ISDN card, and the Asterisk software. Seven companies had laid their hands and minds on it. Sounds interesting? Yes on a weekday it would be.

The client had a server with Asterisk installed on it. But it was not working. Many people, including some regarded as experts in the field, had tried to solve the problems for the last two weeks, but everyone failed. And the client wanted us to get this working before Sunday – at any cost – as there was an important presentation to the CEO of one of the largest telecom companies in India. Does it get any bigger than this! Of course it does. Read on…

To give you a little background, we have an Asterisk based phone system setup in our office. Nirav – our Chairman – had blogged about the problems we faced setting it up with the ISDN line. Sudish – the client – read this post after a Google, and got his ISDN connection working when he followed those instructions. That’s why he called us up, after some failed attempts from many.

Nilkanth, our Systems Team Leader, and a true hero in this story, informed the team about the project. The server was physically located in another city, and we had shell access to it. Nirav had setup most of the things on the Asterisk Server in Magnet. And Ali knew some part about dial plans. Rest of the team knew only the basics of the technology till that Friday evening. Then What?

What else? We jumped in to the battle like brave warriors, The first enemy to tackle was getting hold on more than the basics and doing that with a timer. We had just one plan. “Ground work first” and all the detailed documentation, long process can come in later. Quick analysis seemed the need of the hour and the only trigger we had was a couple of Samosas and chai. Not a bad start at all. The first strike from our team was a detailed check- list of priorities, delegation of all the duties on the list to the best to be working on them. Who knows better than our Nilkanth of his team capabilities?

Now, things started like this for us. Ali left early that evening. Nirav and Nilkanth were working on the server to fix the issues. The Asterisk server was not able to route outgoing calls at that time. There were lots of errors and warnings. We found many files edited by different people. Initially, we tried fixing them. We grouped the errors, set priorities, and got on to them. But the setup was all messed up. It was 6pm by then. Nirav had to leave so, he told us which important files need attention and left. So there was Nil, Arun and me. We tried to fix those errors one by one. But it still did not work. We decided to re-compile Asterisk. We took a backup of current configuration, downloaded the version the client wanted, and re-compiled. We re-configured things. Now we had fixed all the errors – the log files were clean. The outgoing calls from command line were working fine. But, the incoming calls were not coming in – seemed like they were going to a blackhole! Various Voice over IP clients like eyeBeam, SJPhone were showing the Error 403!! It was 3am Saturday by then. We had already decided that no matter what happens, we are going to get it working and only then we will leave for home. Challenges don’t push us down, they make us rise higher!

The spirit of the team and determination to get over the problem helped us to stay focussed. Each of us motivated the other person.

We rebooted the system since we had re-compiled and re-configured, but… we invited more trouble!! The server was unable to detect the Digium card that connects the software with the ISDN line bearing the telephone connections. Oops!!!

This was like adding insult to an injury! We were stuck and no one could suggest how to proceed. We Googled the error. We decided to insert the card in a different slot on the computer. We called up Sudish, who was awake with us and asked him to plug-in the Digium card into another slot. He immediately arranged some person on the site and did exactly as we asked. The machine was rebooted again. And the error still showed up!

After some deep debugging, Nil figurred that the problem is due to Kudzu. Kudzu is a hardware detection agent on RedHat Linux, and it was detecting the card, but not able to understand what it was. We shut off Kudzu on the server, and that got the card back to life!

It was Saturday morning now, and the phones were still not working. Vishal – our CTO – pitched in and we all started working on identifying what’s really causing the problems. Vishal’s jumping in encouraged the team and showed his commitment. He brought a new perspective and was really helpful in checking our assumptions and also researching.

Sudish had a hair pulling time while all this was happening, as the time for the presentation was approaching very quickly. He had all his hopes on us. Did that pressurize us? No we were honoured by the trust.

Sudish was very cooperative in all this… he was with us, offered us his ideas, showed great faith in us and supported us thoroughly. As a matter of fact he was also awake with us to help us. What more can we expect?

We then took a new approach. One closed door did not stop us. We just looked out for another one. We started observing the issues from another angle and brainstormed to analyze a little differently. We divided work: we took turns to download, research, try on our server, communicate with client, think / plan, take care of needs etc. This helped each ones point of view bring new solutions.

Right then, we found that some of the pre-requisites for the operating system were not installed on the server. We downloaded and installed all those hoping that it would fix it! Lady luck was not in our favor, and doing this too did not make the system work! The day went in configuring and re-configuring the setup. Trying out different combinations to see if something would work. The same setup files that worked for us, and a lot of other people, did not work on this particular server.

Sunday morning…and we were all stressed. The client had urgent requirement, the team was working continuously on this from Friday morning, putting their best efforts. Vishal was with us and Nirav in the loop. Nirav actually logged onto the server remotely during break timings of his seminar and tried to solve the problems. Unfortunately, he could also not solve it. This was really getting hot now!

Sunday morning, we spent half an hour discussing the problems and possible solutions with Nirav. He encouraged the team and showed his support. He told us that he will get back to us on chat by evening. But I had some intuition and I told him… “hopefully we will tell you that things are working.. by evening”.

By late morning, we were getting close. We had fixed most of the problems. We were now able to receive calls on the server. We could make outgoing calls from the common line and the interactive voice response system was working too. But the extensions were not registering with the server yet. We re-considered everything. We had a very good dial plan, we had all files configured correctly. We had provided all dependencies to the kernel. So there was very little chance of any problem with all that. Now the only thing that could have some problem was – well, Asterisk itself!

We planned to install the latest version again. We compiled Asterisk 1.2.13 keeping the configuration and the other libraries files intact. Vishal started eyeBeam software phone client and to his surprise, it registered! Catching hold of his excitement, he tried it again. And it worked again. He tried calling, and that worked too! Within a minute, all of us were on it. And every single thing worked! We checked the incoming calls, outgoing calls and the calls within network.. everything working fine! The final compilation helped us fix the issue. We setup our final extension dial plan. And called the client from the working software phone, and gave him the good news! It was Sunday afternoon, and our afternoon tea was unforgettable that day! Not less meaningfull than a medal that said “GREAT” and “ON TIME”.

And now, of course the moral of the story! First of all, weekends don’t need to be the same, they can be more intersting than you are prepared for. Teamwork, Dedication and Planning can take you out of the most difficult situation. We did well but that would not be enough if our planning was not to do it on time and that made all the difference. So conclusion – we understood the final goal, we chalked the steps to get there. We started as a team and finished as a team. We left each work to the best in the team. We enjoyed the challenges. We went back to the drawing board when one option did not work. We assured and updated the client at all times to find more faith and trust each time.

I think that’s one rocking weekend one can have, I could not learn and experience more in 48 hours. Waiting for such interesting weekends ahead!

(People in the photo above, L to R: Ali, Jayesh, Nilkanth, Arun, Ameya. Thanks to Prakash for the creative work!)