We first noticed that textburst pages would occasionally freeze a few days before before Christmas and released a temporary fix that solved it for 99% of users.

Unfortunately as with most temporary fixes it wasn’t 100% effective so we’ve been investigating the root cause since getting back from the holidays.

After many, many hours digging through tens of thousands of lines of log files (it’s amazing how much logging a debug mode can create) and tweaking code here and there we at last found the cause yesterday afternoon. And two lines of code later it’s fixed.

For the technically inclined here’s a little more information:

Textburst runs on four web servers, these are set up as two load balanced pairs, one in Derby and one in Manchester. To ensure users stay logged in when switching between servers we store our session data in a MySQL database.

Our MySQL based session store is a slightly tweaked version of the MySQL session state provider which is in turn based on the Microsoft sample session state provider.

Unfortunately it turns out both of these have a bug when handling concurrent connections.

How things are supposed to work

When a page is requested it should should check to see if the session is currently locked by another page, wait until it’s free and then obtain the session data, along with a lock if the page wants write access and then increment an internal Lock ID by one which is used to make sure the same request is locking and unlocking the session.

What was actually happening

Requests that wanted read/write access to the session worked exactly as expected, however read-only sessions missed the all important step of waiting until the session was free and simply incremented the Lock ID. This meant that if a read-only request was processed in between the lock and unlock phase of a read/write request it would never be unlocked causing the next read/write request to freeze.

Luckily the ASP.NET session code is designed for this eventuality and after 110 seconds the stuck lock is forcibly removed, unfortunately our users were stuck waiting for this to happen while everything appeared to be frozen.

Our fix

We’ve updated the session provider to always wait until the session is free. To help out others with similar problems we’ve added this as a comment on the Microsoft sample and will be filing a bug report with MySQL shortly.

(But not for you recruitment consultant. Sorry, but you’re not in our plans, so don’t contact us about this.)

Our SMS API is shipping faster than ever before. Florence, our healthcare app is gaining momentum in the NHS. Textburst is finding an audience like never before, and we’ve got a completely new app on the cards.

But, arhhhh!

We just can’t find the time to do all the stuff we really want to do. We need some help, and you might be what we need.

What we’re ideally looking for is a web app developer.

You need to have some (not necessarily loads of) relevant qualifications and experience.

You definitely need to know some languages and frameworks, not necessarily these, but things like PHP, c#, .Net, Ruby.

Knowledge of MySQL or similar database will also be looked upon favourably.

We’re also big fans of wordpress and have a number of plugins available to our clients. If you know wordpress then you’re already in our good books.

But most important is your desire to get involved in something. We’re not just tapping lines of code out, we’re making products that people use every day of their working life. Your ideas and desire to make things better are what we want, they’re more valuable than your qualifications.

So if you live and breathe the web, spend your spare time dreaming of quirky web apps, playing with open source software or mashing API’s on programmable web, then we need to hear from you.

You can help us

What’s it like to work at mediaburst?

None of us at mediaburst are here to pass the time of day and collect the cheque.

Coming to work is a pleasure, we enjoy what we do. We want to make our products better, gain new customers and take good care of the existing ones.

We’re not here until the wee hours, you’ve got a social life and we respect that. We also don’t want you to wear a suit and tie unless you really want to, but then again, if you did, you probably wouldn’t fit in.

Our office is completely open plan, there are no walls or secrecy, you can chip in if you like and share your thoughts. We want you to make a difference to the things we do and the stuff we sell, you can’t do that well unless you know what’s going on.

We don’t have accolades like Investors in People, or bureaucratic review procedures. If you want something then ask, if you have a problem, speak up.

Salary

Yes we’ll pay you, but the internet is no place to discuss financial arrangements, let’s discuss in private.

How to apply

Get the ball rolling by sending us an email. Tell us about yourself, include your CV if you like, show us your twitter or facebook or last FM or Google+ or whatever you’re on or into.

Send us a link to any projects you’ve been involved in or sites you’ve developed.

And, yeah, let’s talk.

hello@mediaburst.co.uk

As we were re-writing it we took the chance to make a few other changes too, don’t worry though, if you’re already using the API we haven’t changed anything, all the old URLs and parameters will still work.

Url Changes

From now on all API addresses are on api.mediaburst.co.uk rather than sms.message-platform.com, we feel it makes more sense as you can instantly tell it’s the mediaburst API.

All examples now use SSL for added security, you’re still able to turn it off though.
We have no plans to disable the old sms.message-platform.com.

So the URL for sending through our simple HTTP API becomes https://api.mediaburst.co.uk/http/send.aspx

Delivery Receipts

We’ve renamed a few of the merge parameters for consistency, when you send a text message you use the To parameter for the mobile number, and now the delivery receipt will too. Don’t worry though all the old names still work.

Simple HTTP API

We’ve clarified which features are available through the HTTP API. If you want advanced features they’re all available through the XML interface.

Our new simple parameter list now consists of:

• Username
• Password
• To
• Content
• From (Optional – Message originator)
• Long (Optional – Allow up to 459 characters)

Email to SMS address changes

Again to tidy things up, we’ve changed all email to SMS email addresses to use mediaburst.co.uk rather than message-platform.com.

The new addresses are:

Prior to these plugins the applications we’ve developed rely on the sender of the text message to input the mobile phone number, just as you do when you’re sending a text message from your mobile phone. You, as the sender, type in the recipients number.

This is generally foolproof because the sender has an inherent understanding of the number format required to send the text message.

With an eCommerce plugin we have another party to consider. The purchaser or “receiver” of the text message is the one who inputs their mobile number.

And they do so without any knowledge of the format the store owner needs to actually send the text message.

e.g. ask someone to fill in a form with their UK mobile number and you’ll get things like:

07741234567

07741 234567

07741 234 567

447741234567

447741 234567

+447741234567

44(0)7741234567

and so on, I think you get the picture.

Couple this with the international reach of most eCommerce applications and you’ve got people from all over the world inputting mobile numbers in all sorts of formats, and both the purchaser and their store owners expecting to receive text messages.

The answer is relatively simple, we need to convert any mobile number into a valid international format. That way we’ll have consistency in the data and our SMS API can route the messages to the appropriate country.

Here’s how we currently do it:

To do this you’ll need a table of international dialling codes. We use one hacked together from Wikipedia. Here’s the PHP code.

1. Strip out any leading zeros.
2. Remove any numbers within parentheses
3. Strip any spaces or non numeric characters.
4. Take the country of residence of the purchaser. If the purchaser didn’t input a country then use the store default, and look this up on the table of dialling codes.
5. If the mobile number starts with the dialling code then great, you’re good to go.
6. If not then add the dialling code to the start of the number, and then your good to go.

We are aware that this could be improved.

For instance, if the number starts with a ‘+’ then we could take it that the user has already supplied the country code.

We could also do this if the number starts with ’00′. That is for the UK though, other countries have other codes to dial an international number.

We will be looking at improving the process as we go on.

Preamble:

It’s very basic really, you set up Contact Form 7 to send and email to mediaburst’s email to SMS function and it’s automatically converted to SMS and sent to the relevant network/ mobile phone.

I’ll assume you have Contact Form 7 installed and have a mediaburst SMS API account. If not, then you’ll need to do those tasks first.

The Setup:

So here we go, Contact Form 7 SMS, you’ll need to include

• The destination mobile phone number in the “To” field as part of an email address as follows: 07741234567@sms.message-platform.com
• Your email address in the “From” field. The one you signed up to mediaburst with. We need this to identify the incoming email.
• Your mediaburst API password in the “Subject” line.
• The text message content in the email body (ideally less than 160 characters).

You can see this in action here.

I’ve used the Mail (2) section of Contact Form 7 because I only want summarised information sent via SMS to my mobile, the full contact form submission is still sent via email as usual.

And the second answer is also “Yes”. Mediaburst can send the message to just about any country in the world, just make sure you prefix your mobile number with the relevant international dialling code.

And that’s how to set up Contact Form 7 SMS, couldn’t be easier.

In theory you could concatenate 255 SMS to produce one long SMS, but in practice this is likely to crash a mobile phone, incur significant costs, and annoy the recipient.

Our statistics show that most long SMS contain less than 306 characters (the limit of 2 concatenated SMS), and very few exceed 459 (3 concatenated SMS). For this reason both our web application textburst and our SMS API allow you to send up to 3 SMS that can make up a single long SMS.

If you’re looking for a more in depth technical discussion then refer to our article on Concatenating SMS. It covers the technical requirements and use of the UDH (user data header) to form the long SMS.

We’re not just talking about getting notified by SMS when you receive new tickets, you can configure a wide range of events to trigger the SMS. Maybe changes in ticket status or when a ticket is assigned to you. Pretty handy if you run a small company and can’t be sat in front of a screen on a permanent basis.

I’ll run through a basic set up here, but if you need any assistance just give us a call.

I’ll assume you have a Zendesk account, if you haven’t already you’ll also need a mediaburst SMS API account.

Zendesk SMS Setup

First you need to setup the target in Zendesk:

Under the ‘SETTINGS‘ menu item select ‘Extensions‘.

Click the tab ‘Targets‘ and then ‘add target‘.

You’ll be given a load of options, you want the one titled ‘URL Target‘.

You now have a set of options that looks like this:

Title: We called ours “mediaburstsms” but feel free to call it what you want.

Url: We’ll come back to this in a second.

Method: select ‘GET’

Attribute Name: type the word ‘Content‘ into this box.

Url: The Url field is where the most important info goes including:

• your username and password for the Mediaburst SMS API, and
• the mobile phone number where the notifications will be sent to

You need to copy and paste the following line of code, but replace some parts with your security information. Don’t worry you don’t need to be techy, just read through the code and replace:

YourMediaburstUsername

YourMediaburstPassword

YourMobilePhoneNumber

Replace these words only, nothing before or after them, don’t replace any ‘=’ or ‘&’, literally just those words above.

Here’s the code:

http://sms.message-platform.com/http/send.aspx?Username=YourMediaburstUsername&Password=YourMediaburstPassword&To=YourMobilePhoneNumber&From=Zendesk

Once you have put this code in the Url field you should be able to select ‘Test target‘ and you’ll receive a test SMS.

Save the settings before you click away from this page.

Now you’ll need to set some triggers i.e. events that will give rise to an SMS being triggered. You’ll find them in Zendesk in the main menu under ‘MANAGE‘, then ‘Triggers‘ and ‘mail notifications‘.

I set up a trigger to notify me when we receive a new ticket. To make the information more useful I’ve included the requesters name and the subject field in the SMS.

Here’s a screen shot of my trigger configuration, and the resulting SMS.

And that’s it, I accept the URL mechanism isn’t the most user-friendly for non techies but it’s flexible and, as I said above, if you need any help then give us a call.

Here’s how to add in an originator with a couple of simple moves:

Decide on your originator:

1. It needs to be 11 characters max if it’s an alpha, or 12 max if numeric.
2. You can send from either a name, for instance a company or promotion name, or a mobile number if you want to get replies.
3. We advise not to use spaces or punctuation in your from field, some handsets don’t read them very well.

Check out additional information on originators if you are sending SMS internationally.

Add your originator:

1. Enter the originator into your request, using the ‘from’ parameter
2. Here’s a sample HTTP request using “mediaburst” as the originator:

http://sms.message-platform.com/http/send.aspx?Username=my_user&Password=my_pass&To=441234567890&From=mediaburst&Content=Hello

And that’s it.

Setting a default originator:

If you’re feeling tired of all that coding, you can ask us to set a default originator for you. The default will be used if you don’t complete the ‘from’ field in a request.

Just email hello@mediaburst.co.uk and we’ll sort it out.  You can override the default by specifying the ‘from’ parameter on those occasions you’d like your text to be from something different.

All in all, easy to set, and easy to change.

It’s a mashup of both electronics and a C# application that accesses the mediaburst SMS API. The result is a text message being sent to his mobile every time the door is opened. Simple, but effective.

You can read the full details and spec on Damien’s blog

If you know if any creative uses of our SMS API then get in touch and we’ll do a write up.

It certainly indicates we need more information on the website about our international credentials, but while we prepare something here’s a taster:

Basically sending internationally is as easy as sending to UK destinations, you just need the international prefix (e.g. 44) on the mobile phone number. Line your messages up and watch them deliver round the world. They’ll be just as happy zipping down the road and to the next city too.

Get the code

There’s plenty of useful stuff in our SMS API code kits that will help you with integrating. There’s C#, PHP, Perl, ASP plus a Ruby Gem if that’s your thing.

If you need expert help, we’ve got that too. Our techies have years of experience.  They’re also the guys watching your back and maintaining our platform 24/7.

A couple of tips for you

1. Some international networks block alpha originators or company names.  Use the ‘from’ field parameter to set a numeric originator when sending outside the UK.
2. Just use the international country code. There’s no need for the ’00′ or ‘+’ with our SMS API

Any questions, just let us know, always here to help.

How does it work?

We simply maintain an access database against your account and block anything outside of your named range. It might not be the most exciting thing, but it could be crucial.

For an added level of security, you can also set your system to allow our named range of IP’s for the mediaburst SMS API, which are as follows:

CIDR                                      Netmask                              Address Range

89.248.48.192/27              255.255.255.224                89.248.48.192 – 89.248.48.223

217.20.43.96/27                255.255.255.224                217.20.43.96 – 217.20.43.127

How do I lock it down?

Just send us an email to hello@mediaburst.co.uk with your account details and your named IP ranges and we’ll do the rest.

But Paul recognises that not everyone has email, and wanted to update his customers while they were on the move, via SMS.

Originally Paul was logging into a web service to send text messages but it made more sense to send directly from Capsule, hence looking at integrating with mediaburst.

Using the HTTP interface of SMS API he’s got Capsule CRM to send out SMS updates. He’s even produced this step by step guide so other Capsule users can follow in his footsteps.

Paul was very complimentary about mediaburst, “Luckily I found mediaburst….I am now able to send my customers SMS updates within Capsule CRM, something I never dreamed of”.

All messages regardless of coming through the SMS API, from Textburst, or Outlook Mobile Service will be dealt with in this manner. You don’t need to change a thing.

This is a big step forward in deliverability to the USA. The shortcode 43704 is fully registered with the Common Short Code Administration so complies fully with their network and regulatory requirements.

North America has some unique complexities in their SMS industry that mean delivery to mobile numbers in the USA can be problematic. Before this upgrade we could deliver SMS to the USA but had to disclaim the service, which, I admit didn’t feel comfortable.

In addition to our discomfort we are also receiving a greater demand for messaging to the USA, and, have a product in the pipeline that will squarely align our services with a large body of software developers in the USA.

The general problem was messages were classed as arriving from an international destination meaning a recipient of the message could be charged (some pay for SMS received in the USA) the rate of an international SMS.

Not only that but all machine to handset orientated messages in the USA should come from a numerical number (rather than an alpha numeric that we use in Europe) so that the recipient could reply to opt out of receiving further communication.

All our SMS to the USA came from a numerical long number but that wasn’t actually a proper reply path for opt outs.

With this upgrade we are now in the fortunate position of delivering on a dedicated and registered short code, we deliver via a fully registered carrier based in the USA, and can process the opt outs as is the requirement.

A big step forward indeed, if you have any questions at all about SMS delivery to America then get in touch.

The latest release of fmSMS puts the mediaburst API on their SMS Gateway list.

Millions of people use FileMaker to manage information on Windows, Mac, and the web. With the fmSMS add-on you can send and receive text messages direct from FileMaker.

It’s more than a basic integration, fmSMS supports single and bulk SMS sends, alphanumeric sender ID, delayed delivery and delivery receipts. It’s a complete package to get text messages direct to the mobile phones of your Filemaker contacts.

In the words of fmSMS:

“We know FileMaker is a great Customer Relationship Management tool so it makes sense to integrate the ability to communicate via SMS within your existing FileMaker solution.

SMS is a great means of direct communication with customers, staff, suppliers, students. It’s personal, quick and more immediate than email/fax/letter.”

This first demo was a simple affair, we manually copied the page HTML, output by textburst, into a few new files and then ran them through a shell script which called sed a dozen or so times to replace anything that didn’t work without the full application behind it.

Since putting the new design live we felt we needed to show off all the features to prospective customers. What really surprised us was just how much work is involved to create a fully featured demo.

Our requirements seem simple at a glance:

• Look and feel like textburst
• Include all major textburst features (except payments)
• Be easy to update
• Don’t really send text messages

For various internal reasons the demo needed to run on the same server as this website, this left us needing to run a C#, ASP.NET web application on a server running PHP and Linux. To make this work we came up with the following steps:

1. Log in to textburst on our development platform
2. Grab a copy of all the relevant pages
3. Run some standard find and replaces to remove ASP.NET specific content
4. Add PHP include files at the top of each page
5. Add in page specific PHP includes
6. Remove any development data
7. Copy this altered page and any includes in to a directory ready for release
8. Create some demo contacts and sends using PHP Sessions for display to users

Just over 800 lines of perl later we have a wonderful jumble of regular expressions and a dozen or so php files that make textburst into the simple demo you can see today.

By automating this step we can just re-run the perl script each time we do a release keeping the demo perfectly up to date and preventing any surprises once you’ve signed up.

We do still have to create page specific include files for some of the pages to show how the product actually works. For example, we add an include to the send message page which displays either a sent successfully or failure message depending upon whether all the form fields were completed. It also adds the sent message to your session so it shows on the sent messages page.

This is a longer term project which can be broken down to smaller parts, the first of which has just been released.

This first part is a bug fix where if we can’t deduct credit or charge you for the messages then we provided an “insufficient credit” error.

With this release you will now receive “internal error”.

If you do receive “internal error” then we recommend you retry after one minute.

You can read more detail about error handling on this page of the SMS API Documentation.

Personalised originators let you send bulk SMS messages from textburst with the name of your company or promotion.

The originator can be up to 11 characters but we don’t recommend either punctuation or spaces as some network providers don’t support them in the originator field.

In textburst you can go to Settings in the top right hand corner and select ‘Originator Settings’.

Update with the word you would like to use for the originator and click save.

The next time you send a broadcast, you have the option to select the originator you’ve chosen.

And don’t worry, if you’re stuck for ideas on what to send out to your customers then our list of what to do with SMS will provide you with all the inspiration you need.

One of our SMS suppliers recently changed the encoding on MO messages from Latin1 to GSM as it supports a wider range of characters. After quite a bit of searching the web we couldn’t find any examples of how to work with GSM encoded text in C# so we wrote our own.

To keep things simple we’ve inherited from System.Text.Encoding creating a Mediaburst.Text.GSMEncoding class.  Characters are encoded using a simple one to one mapping of GSM bytes to Unicode character.

We’ve just released it on github under the ISC license, one of the few open-source licenses written in simple clear English, so feel free to grab a copy and use it in your projects.

How to use

We use the following to convert GSM encoded text to UTF-8 on a web server.

The utf8Enc.GetBytes(body) that looks a little out of place is needed as the server assumes all GET and POST parameters are UTF-8 encoded.

string body = "Some GSM encoded text";
Encoding gsmEnc = new Mediaburst.Text.GSMEncoding();
Encoding utf8Enc = new System.Text.UTF8Encoding();
byte[] gsmBytes = utf8Enc.GetBytes(body);
byte[] utf8Bytes = Encoding.Convert(gsmEnc, utf8Enc, gsmBytes);
body = utf8Enc.GetString(utf8Bytes);

Caveats

The encoding functions don’t currently support the EncoderFallback and DecoderFallback error handlers for invalid characters or follow the Microsoft convention of replacing an invalid character with a question marks. Invalid characters will be ignored and missing from the converted string.

Get the code

Download from github

We do provide some excellent Delivery Status information in our Online SMS service textburst and even more if you send via our SMS API. By understanding and using this data you can cut down on undelivered messages and make some good cost savings too.

SMS API

Users of the API can get some pretty detailed information on the delivery status of each message, the full table is in our SMS API documentation.

They 2 key parameters to be concerned with are DELIVERY_STATUS and ERR_CODE.

If under DELIVERY_STATUS you receive UNDELIV or REJECTED (undelivered or rejected) then it warrants further investigation using the ERR_CODE parameter.

ERR_CODE numbers 5 or 9 indicate no matter how many times you try you simply cannot send a text message to this number. Most likely because the number has been cancelled or less likely the phone does not support text messaging. If you get a 5 or 9 we suggest you remove this number and try other means of contact.

ERR_CODE numbers 4, 6, 7, or 10 indicate a Temporary error and chances are it’ll clear given time. We suggest you hold off sending to this number for a week or so, maybe they are on holiday or having technical difficulties with their phone.

It’s also likely you’ll see some of the less decisive error codes including the unhelpful network error code of “unknown – no details provided by the network”.

By doing a periodic review of all your undelivered or rejected messages you can help reduce these errors. Look for the number of occurrences you tried to send to any one given number, if it’s always failing for whatever reason then chances are this is no longer a valid contact number for your customer.

Textburst

If you’re using textburst then the error code information is summarised and we suggest a periodic review of the information to identify bad data. Warning: you might need someone handy with a spreadsheet.

Once you have logged into textburst go to Sent Messages.

Click the small pie chart next to each batch of sent messages.

Then select “List View” in the main heading.

You’ve now got a list of each message, the mobile number and delivery status. You’ll notice on the right hand side an option to download the complete information to a spreadsheet.

We suggest downloading all your sends and searching for those numbers that regularly or never deliver, using this data you can either delete contacts or follow your own path to correct the data for future sends.

I hope this is helpful, if you need any more information or guidance then just ask.

This financial year we set ourselves a challenge to reduce our office energy bill by at least 25%. In terms of electricity, the largest user in the office and the first target was our comprehensive development and test environment of 10 ‘old’ servers.

But how do you create a development and realistic test environment for a dual site clustered Bulk SMS platform using fewer servers than the production service? The answer is with two high spec servers and, as you already know, hardware-assisted virtualisation.

To make the fast servers we simply fitted new motherboards, quad core CPUs and fast RAM into two existing chassis’ saving the cost of buying new cases, power supplies and drives. This left us with 8 redundant servers and a few leftover components.

Electricity

The old development and test platforms spanned 10 servers using a total of approximately 1.8kW direct power and air conditioning at an estimated 2kWh continuous consumption costing about £3,662 per annum in electricity and taking up over 24U of rack space.

Our new virtualised platform is 2 servers at approx 500W total, saving 1.4kW or approx £1,350 per annum plus an estimated saving of £964 per annum on air conditioning.

Space

Old setup = 10x servers @ 24U
New virtualisations = 2x servers @ 6U
Therefore we’ve cleared out half a rack of heat and noise

New equipment

Total £1,200, however we also managed to sell off the old servers on ebay for approx £600, leaving us only £600 out of pocket.

Licensing

Thankfully, not an issue as we already have the platform covered and the two host servers are using the free VMWare vSphere Hypervisor.

Estimated total savings

First year £1,714
Second year £2,314

Now, time to go find a more energy efficient kettle!

What’s in this release?

This milestone release doesn’t contain any major new features for you, our users, it’s all about making our lives easier at mediaburst and allowing for future changes.

We’ve changed to .NET 4

Microsoft released version 4 of their .NET framework a couple of months ago, along with it came Visual Studio 2010. Although there are no groundbreaking changes in this new Visual Studio it solves various day to day niggles.

Code rewriting

We’ve re-written some internal code that passes messages around between systems. It’s just housekeeping really, making use of the improvements between versions 2 and 4 of the .NET framework. The biggest change is the switch from ArrayLists of messages to strongly typed Lists using the new support of covariance and contravariance in generic types. This change allows the compiler to check what we’re adding to the list at compile time, rather than run time, hopefully avoiding any mistakes.

No more image messages

We’ve removed support for Nokia and EMS image messages. They’re small black and white bitmaps which look awful on modern phones, a much better choice, with wider handset support, is MMS. Processing these took large chunks of bespoke code, without them our sending scripts are so much easier to maintain as there’s only one set of logic.

Updated logging

And finally we’ve tweaked our sent message logging to make a number of internal reports run off a single database table. We currently have to link two databases on different servers if we want the same level of detail.

Very simply, the plug-in provides a gateway so you can receive Nagios SMS alerts and notification.

It uses the Nagios pager confirmation to send SMS through our HTTP Gateway.

You can see more details on Nagios’s site and we should appear on the Nagios official plugins directory pretty soon.

Big sends, instant response.

Rather than making you wait around while we process large bulk SMS sends, the send page now responds instantly. 

We now process the send in the background. It means you can keep working while the send processes.

Manage scheduled messages

In the past if you wanted to cancel scheduled message you had to call us and we’d do it for you. 

You’ll now find a list of all scheduled messages within the application and can cancel them up to a few minutes before they’re sent.

Up to date groups

If you schedule a message to a group we’ll now pick up the list of group members at the time of send rather than schedule.

This means you can continue to manage your groups and the send will be up to date with your latest contact list.

Additionally, to make this change work we now deduct message credit when we process the send.

Faster uploads

We’ve made a few behind the scenes tweaks to how uploads are processed.  In our tests an upload of 50,000 contacts now completes 10 times quicker than before.

Bigger uploads, bigger sends

There used to be a limit of 10,000 people in a single upload or send. 

With the speed improvements above we’ve removed this limit and so far have tested with groups of 100,000 contacts with no speed issues what so ever.

Summary

So there you have it hopefully you’ll enjoy the benefits, but as always contact us if there  is something we can improve.

This work follows on from our network upgrades in Derby last week and will bring our network fully up-to-date.We’ll be starting mid morning (around 10) and should be finished by early afternoon.

What we’re doing

We’re upgrading our remaining Cisco PIX firewalls to shiny new Cisco ASAs.

We’re setting up a backup link between our sites to keep your data safe.

And finally we’ll upgrade our Manchester backup database servers to make sure they can always keep up with the load.

Technical details

Our Manchester site is on the 217.20.43.96/27 network (217.20.43.96 through to 217.20.43.127), whilst we do the work you won’t be able to reach any servers running on these addresses.   Our Derby site will be up and working fine on the 89.248.48.192/27 network (IP addresses 89.248.48.192 to 89.248.48.223).

This afternoon we’ll take the Manchester servers out of service by removing the DNS records, we’ll then change them back once everything is tested and finished.

If you login to any of our products you can see which site you’re currently using, our Manchester servers always have a number 3 near the start such as textburst3.mediaburst.co.uk, Derby uses a 3 instead.

If you have any questions give us a call on 0845 050 0000 or drop us an email.

While Derby is turned off  we’ll also be upgrading our network and servers to make sure we offer you the best service possible.

We’ll be starting mid morning (around 10) and should be finished by mid afternoon.

What we’re doing

We’re upgrading our Cisco PIX firewalls to shiny new Cisco ASAs, they’re considerably more powerful and should make our life much easier.

We’ll also install new servers for our mobile marketing services, the old ones currently work fine but we don’t want to take any chances.

And finally we’ll upgrade our backup database servers to make sure they can always keep up with the load.

Technical details

Our Derby site is on the 89.248.48.192/27 network (IP addresses 89.248.48.192 through to 89.248.48.223), you won’t be able to reach any servers running on these addresses while we do the work.  Our Manchester site is on 217.20.43.96/27 (217.20.43.96 through to 217.20.43.127) these addresses will all work fine.

On Monday afternoon we’ll take the Derby servers out of our DNS so that our service addresses only point to Manchester, we’ll then change them back once everything is tested and finished.

If you login to any of our products you can see which site you’re currently using, our Derby servers always have a number 5 near the start such as textburst5.mediaburst.co.uk, Manchester uses a 3 instead.

If you have any questions give us a call on 0845 050 0000 or drop us an email.

For sending messages, our SMS API will automatically take care of the settings need to do this as long as the ‘Concat’ parameter is set when sending the message. The Concat parameter can take the value of 1, 2 or 3, and indicates how many SMS you are willing to use to attempt to send a message.

However, if you have messages which are sent from phones to your application (MO Messages), an understanding of how concatenation works can be useful if users are likely to send messages over 160 characters.

User Data Header (UDH)

The SMS that make up a concatenated SMS are related together by using the User Data Header (UDH) of an SMS. The UDH is a collection of bytes which can be put at the start of the SMS content. It can be used to control what happens to the rest of the content. For instance, it was used to send the older style of Nokia picture messages.

To indicate that the content contains a UDH, a flag on the SMS called the UDH Indicator (UDHI) must be turned on. This tells the phone that it must separate the UDH from the rest of the content.

Format of an SMS with a UDH

The phone separates the UDH by reading the first byte of the content. The number in this byte is the length of the rest of UDH and is called the User Data Header Length (UDHL). The phone then knows how many bytes make up the UDH and can separate it from the rest of the message.

As already stated, a UDH can control various things and so can contain various commands. These commands are called Information Elements (IE’s). These IE’s always take the following format: an Identity Element Identifier (IEI) followed by the Length of the IE Data (IEDL) followed by the IE Data (IED). A UDH can contain 1 or more of these IE’s.

UDH’s for Concatenated SMS

For concatenated messages the IE’s that we are interested in are the ones with Identifiers (IEI) of 00 or 08. These indicate that the rest of the message is a concatenated message.

Let’s take an example of using the 00 Information Element Identifier.

The 00 IEI always has an IEDL of 03. That is, there are 3 bytes in the IE Data section.

The first byte is a reference number. This reference number must be the same for all SMS that make up a particular concatenated message.

The second byte is the total number of SMS that go to make up the concatenated message.

The third byte is the part number that this particular SMS is.

So if we have a message that is split across two SMS the bytes in the SMS may be:

First SMS: 05 00 03 A6 02 01 .. bytes that make up the first part text ..
Second SMS: 05 00 03 A6 02 02 .. bytes that make up the second part text ..

The 08 identifier works the same way as the 00, but instead of a 1 byte reference number, it uses a 2 byte (16 bit) reference number. Therefore its IEDL is 04, there are 4 bytes in the IE Data section. These are as follows:

The first and second bytes make up the 16-bit reference number.

The third byte is the total number of SMS that go to make up the concatenated message.

The forth byte is the part number that this particular SMS is.

So another two SMS examples could be:

First SMS: 06 08 04 F4 2E 02 01 .. bytes that make up the first part text ..
Second SMS: 06 08 04 F4 2E 02 02 .. bytes that make up the second part text ..

The API

As mentioned at the start of this article, our API will take care of all of this for sending messages. All you do is set the ‘Concat’ field to maximum SMS that you are willing to use to send a message.

For receiving messages, our API will forward through the individual SMS that make up a concatenated message. The UDH is already removed from the content and is present in the UDH parameter as the hex encoded bytes. The rest of the content is placed into the Payload parameter.

For most single SMS, a UDH parameter will not be present and so the parameter will be blank. But for concatenated SMS, the receiving script can check the UDH for the 00 or 08 IEI’s to allow it to piece together the original message.

Gone are the days of huge pdf files and big print runs. The Online version is well structured so you can easily find the sections you need. If you need something specific then there is also a handy search facility.

To make life easy there is a large bank of example code covering all the different integration protocols.

Additionally, the online guide covers both SMS and MMS.

Upload Contacts

Our largest source of customer queries surrounds uploading bulk SMS contacts. This is in part because the current page is not exactly what you’d call intuitive. (click the image)

The sheer length of the page scares users. The instructions are at the top, perhaps logical, but most users forget the latter steps by the time they have completed the first stages. The “select fields” and “groups” sections in particular are confusing.

We’ve made a few tweaks and come up with this draft which we believe will make the process much smother.

You’ll notice the page is now much shorter with clear instructional copy to guide you through the process.

To de-clutter the page we’ve removed the instructions of how to convert excel to CSV. These is no reason to force you to read these instructions, we’ll include a link to a popup window so you can choose.

We’ve also introduced a preview sample of the uploaded data for you to confirm before committing. And, importantly, you’ll now be able to continue working in Textburst, sending and receiving SMS, while the upload is processing. The actual upload will process in the background, no more waiting for the screen to refresh.

A much improved process, I hope you agree?

Timeout Settings

We currently time out a session on 20 minutes of inactivity.

On this point we’ve listened to you, we’ve heard your requests, we know you don’t like this, so we’ve transferred power back to you:

You log in

You stay in

You log out.

Improving the “To” Field

As web technology improves it’s important to include those developments which add to the usability of our products. To make the send message page more fluid the inclusion of a clever bit of coding will certainly help in the “to” field.

I’ve struggled to describe this in a non techy manner but I think this series of images speak for themselves.

On the send an SMS message page, you have the “To” box that you can type numbers, names, or groups into:

As soon as you click in the box a menu appears:

Start typing and the Ajax functionality will automatically find relevant SMS contacts and groups, simply click the contact or group you were looking for.

Personalise

Finally, you may have noticed the addition of “Personalise” in the image above.

The current version of merging fields in a text message is cumbersome and hence frequently overlooked.

Recipients of SMS messages will undoubtedly respond better to a personalised message and hence why we have drawn more attention to this functionality.

As you are typing a message, use your mouse to click the relevant field to insert into the message.

Simple, effective, and helping you improve communication.

Until January

Hopefully you’ll see your feedback is being taken into consideration and an improved product is just around the corner, scheduled for release early January.

The UK mobile networks all use the GSM standard, and as such, a standard text message is limited to the GSM character set.

An SMS can contain up to 140 bytes. The GSM character set is encoded using 7-bits, rather than the usual 8-bits that make a byte. This means there can be 160 characters in an SMS.

This 7-bit limitation means only 128 standard characters can be encoded. The GSM standard gets round this by also having the Extended GSM character set. These are another 10 characters which are actually sent by sending two 7-bit characters, an escape (ESC) character followed by another character. This means that 160 ‘£’ symbols can fit in a single SMS, but only 80 ‘{‘ symbols.

If you need to send other characters than those in the GSM character set, then take a look at UCS-2 messaging which allows sending most unicode characters, but as each of these takes 2 bytes it means only 70 characters can be sent per SMS.

So our customers don’t need to bother converting characters into the GSM encoding or escaping the extended characters, our SMS API accepts messages in UTF-8 encoding.

Update: Since posting this we’ve released an article on how to do GSM character encoding in .NET

The tables below show a full list of the GSM characters, standard and extended. They also show the equivalent UTF-8 encoding need to send into our API.

Standard GSM Characters

Extended GSM Characters

In an SMS these are prefixed with the escape character (1B) and therefore take up 2 of the 160 characters of an SMS. They do not need escaping when sending into our API.

In this blog I’ll guide you through how we test and validate a new bulk SMS route.

But the first question that arises is, “does the quality vary?”
As you’ll read below, the answer is a resounding “yes”

…massively.

Stage 1. One message, One Handset

The first stage is to send a single message to a single phone, it sounds incredibly daft but if we can’t do this then we can stop wasting our time or least contact the supplier and get them to sort themselves out.

Can this really go wrong: if the supplier hasn’t configured the account properly, has provided you with the wrong user name and/ or password, has locked the route to a wrong IP address, or provided you with inaccurate API documentation then yes, it can go wrong.

Stage 2. Multiple Messages

Now we send 6 messages, one to a handset on each UK network. They should all arrive at approximately the same time or at least seconds of each other. The message should be the same on all the phones and the header/ originator should be the same.

What can go wrong here: In the past we have seen messages delivered to all but one phone, we have seen SMS headers scrambled on certain networks, characters changed in the SMS message body, and even routes that can’t deliver to Nokia handsets.

Stage 3. Character Encoding

Now it starts to get tricky, we send the complete GSM character set to each of the 6 phones. Some say this isn’t possible. But that’s a lie, its depends on which routes they use, which networks you send through. For example, one of Vodafone’s cheaper routes doesn’t process the “|” (pipe) symbol, but you can route through O2 to a Vodafone handset successfully with this symbol.

So we check through the character set to ensure they come through accurately on each phone, sometimes some Greek symbols and other lesser used characters will be replaced, sometimes the message isn’t delivered at all, but at worst, the message isn’t delivered but the supplier returns a delivery receipt saying it was delivered.

This can happen because the supplier or network isn’t able to process a particular character, they simply fail the message, but systems can get confused and think it was delivered.

If we get a good result on this test then we stop here and wait a week.

Stage 4. Repetition

Repeat stages 1 to 3 again. We like to double check things don’t change.

Stage 5. Integration

If all is looking good then we’ll start the integration process. This means configuring our live platform to send a limited volume of real messages.

We’ll send a regular stream of messages for a few weeks but monitor very very closely. We look at delivery rates, latency, we’ll do occasional checks to the in-house test phones.

We’ll always allow this test to cross over a month end because we need to see a bill for the messages. We need to see our records of messages correlates with theirs. We like to see they are able to raise an invoice promptly and accurately. We want to ensure we can access relevant technically competent staff in a very short time frame, that means they have to answer the phone and emails and resolve any issues quickly. It’s not just about the messages, we look for solid, reliable, consistent suppliers, and this can only be proved over time.

Stage 6. We’re Live!

Increase the volume of SMS through the route.

But we’ll only do this if everything above is satisfied to our exacting standards, and even then we continue to monitor on a regular basis. But regular monitoring is a subject for another blog post…

Having tried a Juniper SRX for a fortnight we were left with the impression that a few features aren’t quite finished yet. The routing side, which Juniper are traditionally associated with, was really powerful and easy to use, but for a firewall it seemed to be lacking a few useful features found on the majority of the competition. These included authenticating VPN users locally if your Radius server is down and assigning IP addresses without an additional DHCP server. From talking to a couple of Juniper experts it sounds like these and many other features are coming soon, unfortunately for Juniper we needed them now.

The Juniper SSG range were almost identical to the Cisco products both in features and price, in the end we rejected these because we had major trouble obtaining some competitive quotes. One major international IT reseller were that frustrating to deal with that we’ve vowed never to use them again.

One of our core requirements was dynamic VPN access so we can reach our servers when on-call and people can work from home. Given that Microsoft have just released Windows 7 and that 64bit operating systems are now being pre-installed on laptops it had to work with both of these. The Juniper VPN client only works on a 32bit OS, we’d have to purchase some third party software to connect from a 64bit OS, Cisco on the other hand have added full 64bit support to their AnyConnect client.

Our final reason is that Cisco were willing to accept our current PIX firewalls as trade-in. Of all the reasons we considered this was least important but it became a small factor once we realised the features were essentially the same. The same international IT supplier strung us along for 3 weeks while trying to sort out this trade-in with Cisco, they then gave up. Our normal, much smaller, UK supplier had it all sorted in an afternoon and managed a much better price too.

These new firewalls should be with us shortly and I’ll post an update once we’re ready to put them live.

When a message delivery fails and the network returns extra information to us regarding the failure, we can now send this information back to you in a standardised way.

The error code parameter is already defined in section 3.5 of the our API. Please contact us if you want this enabling on your default delivery receipt script. Alternatively, if you set this information when you send a message, you just need to use the ERR_CODE parameter. For instance, if delivery receipts are currently set up to be a GET request using the following parameters:

client_id=#CLIENT_ID#&status=#DELIVERY_STATUS#&msg_id=#MSG_ID#

Then the error code can be also passed by changing this to:

client_id=#CLIENT_ID#&status=#DELIVERY_STATUS#&msg_id=#MSG_ID#&err_code=#ERR_CODE#

A large number of our servers run CentOS Linux, and like all Redhat based distributions software is installed through RPM. Unfortunately installing perl modules using CPAN has nasty habit of clashing with the base perl install and causing no end of problems.

To work around this we’ve taken to installing all perl modules through RPM. Some modules are easy to find in third party repositories, but our software uses some less common modules such as Net::SMPP.

The best way we’ve found to convert these elusive perl modules into RPMs is through cpan2rpm by Erick Calder, it’s also a very convenient way to make our own software available as an RPM.

Creating an RPM from a CPAN perl module is extremely simple, just pass the module name as the only command line parameter, it will download from CPAN and build the RPM.

[rpmbuild@dev-linux ~]$cpan2rpm Net::SMPP
...
RPM: /home/rpmbuild/rpm/RPMS/noarch/perl-Net-SMPP-1.03-1.noarch.rpm
SRPM: /home/rpmbuild/rpm/SRPMS/perl-Net-SMPP-1.03-1.src.rpm
-- Done --
[rpmbuild@dev-linux ~]$

cpan2rpm however does have it’s limitations so we’ve added a few extra features to our version.

RPM Pre install commands

Commands that need to be run before you install the package, such as creating users. Defined in Build.PL using rpm_pre or passed on the command line as –pre=”my pre-install command”

RPM Post install commands

Commands to run once the package is successfully installed (These don’t run if it fails). Defined in Build.PL using rpm_post or passed on the command line as --post="my post-install command"

Disable auto-generation of the provides list

By default rpmbuild will automatically create a list of all packages it provides, on occasion this is wrong, so we added a flag to disable this automatic list creation. Passed to cpan2rpm on the command line as –no-auto-prov

Support for Module::Build dependencies

cpan2rpm couldn’t wouldn’t correctly create dependency lists for modules build using Module::Build rather than ExtUtils::MakeMaker, we tweaked the script so these are now listed on the RPM. These come from the requires section of Build.PL

Support for the RPM config flag on files

RPM allows files to be flagged as configuration files within the RPM. This flag prevents RPM from overwriting your configuration files when changed by creating the new version as file.rpmnew. By default we’ll flag anything from the etc_files section of Build.PL

Using John’s earlier Build.PL example we can add in these extra features so that it creates it’s own user and then runs a postinstall script once the RPM is installed (this could setup a database or download updated configuration files for example). These rpm_ tag’s aren’t standard Module::Build features but it will ignore unrecognised tags when building.

use 5.008005;
use Module::Build;

my $build = Module::Build->new (
    module_name => 'ModuleName',
    license  => 'perl',
    dist_author => 'A.N. Other <a.n.other@example.com>',
    dist_version_from => 'lib/ModuleName.pm',
    etc_files => {
        'etc/logrotate.d/modulename' => 'etc/logrotate.d/modulename',
        'etc/modulename.conf' => 'etc/modulename.conf'
    },
    install_path => { 'etc' => '/etc', 'script' => '/usr/local/bin/modulename' },
    rpm_pre => 'if ! id module_user > /dev/null 2>&1 ; then useradd -r -c "Module User" -s /sbin/nologin -d /usr/local/bin/modulename module_user; fi;'
    rpm_post => '/usr/local/bin/modulename/postinstall_script'
);
$build->add_build_element('etc');
$build->create_build_script;

If you use cpan2rpm regularly I’d recommend creating a .rpmmacros file in your home directory to save you having to pass a few of the options. This also allows you to enable the optional GPG signing of packages. My .rpmmacros file is:

%_topdir /home/rpmbuild/rpm
%packager Mediaburst Ltd <rpmbuild@mediaburst.co.uk>
%_signature gpg
%_gpgbin /usr/bin/gpg
%_gpg_path /home/rpmbuild/.gnupg
%_gpg_name my_key_name

If you plan to use cpan2rpm with standard Perl modules from CPAN I recommend you download it directly from the authors site, however I’m also making available our updated version in case you want to use any of these update features.

• Official cpan2rpm release
• Mediaburst cpan2rpm (we’ve added an mb suffix to the executable)
• Patch containing our changes

I’m not the world’s greatest Perl programmer as I normally write C# so if you’ve got any feedback please leave a comment below.

By default, the encoding is set to UTF-8, but it also supports Latin-1, UCS2 (Unicode), or GSM encodings.

Give us a phone if you want to know more.

The obvious choice seemed to be the Cisco ASA range, probably the ASA5510, but in some areas the changes between the PIX and ASA platforms seem quite significant, so we thought we’d look at the competition too.

A little research narrowed us down to two suppliers that could provide all the features we need within a sensible budget, Cisco and Juniper. Cisco are a safe bet, we’ve used them before and you don’t get any surprises, Juniper on the other hand have only recently started targeting smaller networks, most of their products are used in core networks.

The main contenders were:

Cisco ASA5510

The ASA line is a new improved version of the PIX. They run the same OS and have a number of new features such as SSL VPN tunnels.

Juniper SSG140

The Juniper SSG range is based on the products previously made by Netscreen (acquired by Juniper a few years back). They run a custom OS called ScreenOS and very similar to the ASA5510 in all features we need.

Juniper SRX210

Juniper have only recently introduced the smaller models in their SRX range, these run Junos, the same OS as Junipers core routers. From the figures we obtained they appear to have considerably higher throughput and performance then either the ASA or SSG ranges and a number of additional features.

At the moment Juniper seem to be really pushing the SRX line, they’ve got really competitive pricing and free online training including certification, and given that they have more features than either of the other products we’ve going to give them a try.

A friendly Juniper reseller has agreed to lend us one for a couple of weeks while we see if they’re as good as they look on paper.

Once we’ve got our hands on one I’ll add another post letting you know what we think.

We have been investigating the various build/packaging options for perl modules.

Up to now we have been using the standard method of ‘h2xs -AXn Module::Name‘. This generates a template module with a ExtUtils::MakeMaker build file.

We can then use cpan2rpm to create rpm files that we can put on our internal yum repositories.

This allows us to easily install/upgrade the same module across our various servers.

The problem has come when we wanted to have a slightly modified rpm e.g. have the rpm create a standard user, install a binary and/or init script, or create a config file.

The ExtUtils / cpan2rpm combination does not appear to have this functionality. We started looking at putting it in, but it was becoming increasingly difficult to do so.

So we started looking at Module::Build instead of ExtUtls. With this it is easier to make custom builds, and because of this seems to becoming increasingly popular as the Perl module builder.

For instance the Build.PL to enable a config file install into /etc and a logrotate script can be as follows:

use 5.008005;
use Module::Build;

my $build = Module::Build->new (
module_name => 'ModuleName',
license => 'perl',
dist_author => 'A.N. Other <a.n.other@example.com>',
dist_version_from => 'lib/ModuleName.pm',
etc_files => {
'etc/logrotate.d/modulename' => 'etc/logrotate.d/modulename',
'etc/modulename.conf' => 'etc/modulename.conf'
},
install_path => { 'etc' => '/etc' },
);
$build->add_build_element('etc');
$build->create_build_script;

We still need to use our modified cpan2rpm if we need any pre or post install scripts in the rpm. But so far this appears to be working.

If you’ve got any questions about any of the changes we’ve made give us a call or drop us an email.

We’re now happy to say everything appears to be working fine. If you have any questions or queries about the move give us a call on 0845 050 0000.

Here’s a few pictures to show off our tastefully coloured network cables.

We no longer have any services running on the 213.218.217.160/27 range of IP addresses.

Update

Everything is now running fine in our Derby site, we’re not going to transfer over customer traffic until Tomorrow (Tuesday) so we can keep an eye on the log files.

A few pictures of where we’ll be hosting the services

IP Address Range Changes

The development requires clients that restrict access to/from Mediaburst services based on IP addresses to make the following changes:

From 27th June 2009:

Services will start running on the 89.248.48.192/27 range. (89.248.48.192 through to 89.248.48.223)

From 29th June 2009:

Services will no longer be running on the 213.218.217.160/27 range. (213.218.217.160 through to 213.218.217.191)

There will be no change to our Manchester, 217.20.43.96/27, range of IP addresses (217.20.43.96 through to 217.20.43.127)

If you have any questions give us a call.