November 10, 2011: Busier than I-don't-know-what

This past week or so, I've been coding furiously for the site. It seems as though I have one idea, and before I'm done working on it, I have another to try that dovetails or replaces it. It's not big showy stuff, unfortunately, but more quality-of-life type things.

The train of thought kinda goes like this: I have a service for sale. I want to advertise the service (by buying ad spots on other WoW economic blogs/tools and/or Google adwords). But before I do that, there are things that need to be tightened up. And I think of lots of those things to polish before calling it done.

One example is the Youtube videos. I haven't forgotten them, and I fully intend to complete the series of about 8 videos. I've just been busy coding instead of recording. Making those videos is a chore, let me tell you. I'm running Linux at home, and the software I used for the first videos only uses "open" video formats, which most decent video editors and Youtube don't accept. So I've had video conversion issues (which I may solve by using ffmpeg to record next time instead). Then I have to edit the video to take out all my "ums" and rambly mumbly bits that you don't need to hear. Then I upload it, and write a transcription, so it'll have accurate synced closed captioning. Then I find the "chapter" timestamps to put in the video description. I know I don't need to do all that at the end, and that most users won't care to see the video, never mind the closed captioning, but I want to do it right because I can, and I want to have little out-of-the-way links on the various pages that say, "are you lost? here's a video about this page to give you a tour."

New thought: maybe just use javascript and embedded audio right from the page? Hm.

Anyway, you see how this is going. I want to add something that gives that little extra polish, even though it usually ends up as a lot of work.

Wall of text of stuff done recently, in no particular order:
  • The progress bar on items with lots of auctions (think herbs and ore) would linger at that "Getting item price and availability over the past 2 weeks.." step. Now it has accurate(!) tiny little steps as that big step progresses, so you don't stare at a motionless loading bar for too long.

  • Gave another look at Yahoo's Best Practices for Speeding Up Your Web Site. I already had the CDN from two weeks ago, and I've implemented other suggestions months before that, but there was more I could do.

    • In an effort to make fewer HTTP requests, I thought about the page loading process. For items, sellers and categories, when we don't have fresh data in the cache, it shows the progress bar. The progress bar is accurate while loading, and once all the data is collected, it's pushed into the cache, and the page reloads. Then, on the second load, it's a cache hit, and it loads the actual page fairly quickly.

      I went through a bunch of effort so, at the end of the data collection, instead of reloading, it hides the progress bar section of the page, and continues with the actual page with the graphs and everything else. This meant no reloads, and fewer HTTP requests. However, it also meant I couldn't use gzip compression anymore (since, to show the progress bar, I have to flush to the browser frequently, which doesn't permit the use of gzip on my server). I ended up quadrupling the bandwidth used in order to save an HTTP request. Boo. That lasted about a day before I switched it back.

    • I implemented a "wait" in the progress bar display. The idea was, if the page took less than 2 seconds to load, then instead of showing the progress bar, I'll just wait until the page collects all its data and present that. If it takes longer than 2 seconds, then push the progress bar to the browser, and work as it always did. However, that just made the site look laggy (since nothing happened for up to 2 seconds after a link was clicked). That change ended up reverted, as well.

    • I put all the javascript at the bottom of the page, both in-line scripts and those linked in. The hope was that the page would show something quickly after the browser started getting data, then do all the scripty bits afterwards to fill in the charts. It may have helped, a little. Maybe.

    • Minimized HTTP requests by putting the 4 login icons in the top left into an imagemap instead of having 4 files. Hey, it's something, and it's on every page.

    • I used two different domain names for the CDN, to load objects in parallel. I was gonna have a bunch hanging off of but then noscript would cough when you tried to load scripts from that new domain. I decided to keep all scripts on and but load most/all static images from Using the domain also means we don't send cookies with every CDN request.

    • I minified much of the static javascript and some of the dynamic javascript that's in-line in the pages. I tried to minify everything in the HTML that gets sent to the browser through clever use of regexps, but that caused problems and ended up increasing server-side load time anyway.

  • When new syndicated editorials are posted here, they're also broadcast on Twitter at @UndermineJrnl with an appropriate pretty small URL. :)

  • The above change also forced me to finish the move of RSS collection off of my home server and completely onto the main web server. It had a lot of old code and processes from when I manually approved and edited posts.

  • PubSubHubbub! (PuSH) We do a lot with RSS feeds here, and I never looked much into this push technology until this week.

    • For starters, all the syndicated editorials work via pulling RSS. Just under half of the blogs we watch use PuSH to notify subscribers as soon as new content is available. I wrote a small PuSH client into the feed aggregator to subscribe to feed notifications. That way, for participating blogs, as soon as they post, the article shows up here. No lag. Cool stuff. I saw it work for one post already, but not all blogs participate, and it's been a slow news week besides. I think it works.

    • Second, I'd like RSS feeds for logged-in users to be more immediate.. i.e., use PuSH as publishers. Technically, we have thousands of feeds, one for each user, but each feed would only have one subscriber, so I didn't want to use a public hub to announce notifications for everyone when nobody might be listening. Instead, I've implemented a custom hub for the site. Since I can keep track of when a feed has a PuSH subscriber, I can announce directly to the subscribers without wondering if the hub has any listeners.

      Long story short, this should make RSS feeds update more quickly for users. I've had trouble testing it, though, since it's not 100% immediate between when a reader receives a PuSH notification and the post shows up in the feed reader. It's hard to tell if a new post shows up due to the PuSH note or just from a scheduled check. I think it works. I've been trying it on my own account for the past day or two, and will roll it out to all users in the coming days. Again: no big difference or announcement with this, and the users don't have to do anything different, but it should make things work just a little bit better.

    • In anticipation for making the main page RSS feeds PuSH "compliant", I've consolidated them to one URL, instead of separate URLs for each realm. I may make the forum RSS feeds publish to a hub, too.. not sure yet.

I also want to explore perhaps having "holiday" header images. Nothing overdone or flashy, because I know there's much to be said for a consistent, clean interface. However, I thought about seeing if I could pull the holiday garlands that you'd see hanging from doorways in-game from the game files, then overlaying those on top of the title graphic during those holiday seasons. Maybe. It might look like crap, Idunno. I just want to see if I can pull the images first, and go from there.

All this talk about PuSH tech on the feeds made me explore XMPP for notifications once again. This could announce the same stuff you get via email/rss to you directly via instant message, if you use Google Talk or any other XMPP-compatible service. Just started looking into this. Not thrilled with the steep learning curve to automatically send messages via an XMPP service, but perhaps there's a decent PHP library out there for this. I'd host my own XMPP server to send the messages for this, natch.

I looked for a long time at potentially supporting SMS notifications. The problem is cost. The cheapest I could send an SMS is $0.01 per text, and even at that price, I'd lose money with the volume of the market notification users. And, what could you do with SMS that isn't already done with GMail? To act upon a text, you'd have to fire up the game, a web browser, or the mobile AH app anyway. GMail instantly pushes emails to your android phone, so if you have email notifications sent to your GMail, you're done, and with more detail than could be squeezed into an SMS. And if someone suggests provider Email-SMS gateways, I'll remind you that each cell provider has a different address, and they can change, and some of them squeeze in the "from" and "subject" lines (which take up room), and some don't, and it's just a mess. So, SMS is out, again. Use email that your smartphone can read, and/or an RSS reader, which will soon should get even more prompt when I turn on the PuSH system.

So I'm getting a bunch of ideas lately, and generally coding like a fiend. And most folks won't notice. But I'm happy, because in the end, the site just keeps getting better and better. :)

Category: General | Posted by: admin


November 10, 2011, 21:54:20 Xsinthis wrote:

That PubSubHubbub looks cool, and I just installed a plugin for wordpress that should make my blog compliant. Just to clarify cause there isn't a whole lot of documentation I've found yet....I add as a hub?

November 10, 2011, 22:43:56 admin wrote:

You'd use a hub that supports public blogs. Most use http://pubsubhubbub.appspot... because it's free. There's a Wordpress plugin that's popular, as well, that acts as its own hub.

November 14, 2011, 07:47:35 Sapu wrote:

I know of a way for you to be able to send SMS messages easily and free (from your end). Of course, the receiver still gets charged (at least in the US, not sure about cell companies in other countries) and there are a few other limitations like you can't control the subject line and it'll put a blurb at the end of most messages. Definitely not an ideal solution and probably not worth pursuing anyways as you described, but it does exist and is free and reliable (not some shady company).

Message me in IRC if you're interested at all.

November 14, 2011, 14:10:12 admin wrote:

I've seen Zeep Mobile... meh. :/

November 14, 2011, 15:53:56 Sapu wrote:

That's not what I was thinking of but by the looks of it is very similar...

Add Comment

This item is closed, it's not possible to add new comments to it or to vote on it