The Undermine Journal - Development

Yesterday I mentioned I'd remove the GE addon from WoWInterface. I sent the admins a PM asking for the addon to be removed (the method they request). We had a brief chat about why I wanted it removed, and they pointed me to this Java app. It's a command line utility that one could use to build zip files from version control and push them to WoWI, or to push pre-built zips to the site.

A quick glance at the source code looks like it pretends to be a web browser instead of using some upload API from WoWI, but whatever, it works. The Undermine Journal GE will remain on WoW Interface and get updated automatically from now on, yay.

We've had the GE ("General Edition") addon for quite a while. I would update it once a week, and it is a chore. It's one of the few manual processes I have left with the site. I have to run the script to generate the average data, then go to both Curse and WoWI, login, navigate to the update section, and upload the zip file through the web interface.. it's a drag.

My project is different from most addons because I have weekly updates, not in source code, but in the data. I don't have a need for revision control because I don't have other authors, nor is the addon itself so complex that I'd benefit much from having old versions available.

A few months ago I looked into automating updates. Curse has support for revision control as well as uploading through a script. I worked on an upload script, but didn't get it working for one reason or another. I blamed Curse, although now I think it may have been a bug in my code. WoW Interface has revision control for source code, but still forces you to login and click some buttons to pop out a zip file.

Today I gave it another look, since I've not been timely with the GE updates in the new year. I found an issue with my Curse script, and fixed that. I made a cron job for it, and now it should update every week without any thought from me. WoW Interface still doesn't allow you to post zips automatically (which is no fault of theirs, because my project is a little weird) so I've asked to have my addon removed from there. It's not like I got that many hits from WoWI anyway. It was something like 60 hits per week. I'd get ten or fifteen times that from Curse.

Both the realm-specific addon and the market API have had steady success since release. They usually average around 400 hits a day each. Not bad for services that require a login.

Sidenote: Back when I accepted donations, I also had donation buttons on both the Curse and the WoW Interface addon pages. I got zero donations through those sites; all the donations I got were from the pages here.

There's now about 100 minutes of me droning on about The Undermine Journal on Youtube. I finally finished the User Controls and Notifications videos. I intend to have links on TUJ pages that have videos, so people who want help can just click a corner and have the video about that page pop up. I'll have to balance it so it's out of the way for normal users, but it's still visible for people new to the site. I'll think of something.

Fun fact: That chirping you hear in the background of most of the videos is from my birds. I talk, then they talk.

We've often had requests to support non-English EU realms here at The Undermine Journal, and there are numerous obstacles in the way of full support. Some problems are technical, and some are due to language differences. You can visit our forum, with a spiffy new layout, for more details.

Despite these challenges, we persevered, and we are pleased to announce partial support for all non-English EU realms. You'll find them on the EU realm list as you would expect. Regarding "partial support", due to space constraints, we cannot provide the detailed item and seller history that's available on our English-speaking realms. However, the good news is that you still get access to the Addon, the Market XML API, all of our great category pages (including the Deals page), summary item information, and market notifications!

We'd like to watch how these resources are used, and depending on demand, we can explore native-language pages and tooltips, and perhaps scrape together what we need for full auction history support. To kick things off with a bang (true Goblin style), and to thank our patient visitors from non-English realms, E-Mail and RSS market notification events for non-English EU realms are free to all users until the end of the year.

As always, thanks for your continued support of The Undermine Journal.

I've got a number of things on my to-do list for this site, and some already done. Might as well do like in the old days, and post the list here, so I won't forget and can be held accountable. ;)

• Done: global item pages. Maybe I'll throw in a search box so you don't have to pop back to a realm to find another item. Maybe.
• Need to add US realm Azralon.
• Thinking about adding non-English EU realms, but without auction history. About half the features on the site will be inaccessible, but perhaps it's still worth it. Check the forum for more details.
• Think about further instructions about OpenID on login pages.
• Think about having special instructions for the users whom LiveJournal left high and dry.
• Add seller's past auctions (sold/expired in the past 2 weeks) to seller pages
• Add daily-resolution data to item history graphs for 28-14 days before present
• Use same data to add 14-0 day sparklines to category top items pages
• investigate searchbox autocomplete, again
• add stack size multiplication to addon
• continue with the videos, making more and having links to relevant videos on pages here
• opt-in email "newsletters" to subscribers, twice a day, with new syndicated editorials, user-selected category page sections, and maybe notification status
• ask about wowtcgloot.com affiliate links

Okay, for that last bit. Just had this thought like 20 minutes ago. There's WoWTCGLoot.com. They sell, well, WoW TCG loot cards. Maybe I should have affiliate links on TCG item pages to that site. They wouldn't be gaudy banners or anything, just a text link only on TCG item pages that says, "Hey, you're looking at Amani Dragonhawk. You can be crazy and spend real money on this. Check it out." Might make a few bucks on the side, and remind people that there are other ways to get the item (or, hell, other ways to resell it). Good idea? Bad idea?

edit The last bit is done and implemented. It's actually quite useful and not jarring at all, imo.

Here it is, released this afternoon.

My first knee-jerk reaction is to take my ball and go home. I felt the same during the addon nonsense. Luckily, TUJ has grown into a very big ball, so I won't run away with it just yet.

First, the cold reality. This site does use their APIs (among others), and Blizzard would be within their rights to remove my access to them. This site does use their intellectual property, and Blizzard would be within their rights to restrict my use of that. And if they exercise those options, the site would close. That was always the case and that hasn't changed.

I have no intention of open-sourcing any of my code for The Undermine Journal (with the exception of the addon, which already complies with Blizzard's addon policy).

Edit Jan 12, 2012: My Battle.net-facing source code is here.

I have no intention of reverting to a donation (or free) model to support the cost of the site. The majority of the services provided to the public are free of charge, and the fees I do charge are reasonable and easy to understand. The income allows (and encourages) continued support and development of all aspects the site in a way that donations cannot.

I have no intention of obscuring any information that may be considered "negatively affecting players," such as seller history, which is not accessible from within the game.

In short, I won't change anything regarding the above without a direct request from Blizzard. And if they do request such, or if they do remove my API keys, the site will close. I don't need to bend over backwards to help advertise their game and keep their players engaged.

I have a feeling that business will continue as usual. Developers have been waiting a long time for this policy, and it seems more like a legal formality than a change in stance by the API team.

I've always been a fan of Child's Play. This year, with our notifications subscription, I thought about having something like, "donate at least $5 to Child's Play, and you'll get a notifications sub." Or maybe having all new subs within a time period matched with donations, or however it would be arranged.

But then you get to the sticky side of business. Of course, the folks at Child's Play are concerned about having their name used in a fair way, and any such arrangement needs to be approved by them. Understandable. And it helps if the people running such events (edit: people such as myself) were either registered as non-profits themselves, or never touch the money. And tracking those transactions to give credit on TUJ got to be tricky/impossible. So anything slightly complicated I could come up with became really complicated behind the scenes.

And I can't afford even a Silver sponsorship level. :)

So, long story short, we donated $100 from the TUJ account to Child's Play. I encourage you to donate more to Child's Play. If you're a subscriber here, now you know that some of your money went to a worthy cause.

I never really liked Facebook, and this sort of crap isn't helping.

Thinking about disabling new Facebook logins for TUJ. Existing logins would continue to work, but to create a new account here or to add Facebook to an existing account would be disabled. This would force new users to login via Google, Yahoo, LJ or other methods. Thoughts?

This past week or so, I've been coding furiously for the site. It seems as though I have one idea, and before I'm done working on it, I have another to try that dovetails or replaces it. It's not big showy stuff, unfortunately, but more quality-of-life type things.

The train of thought kinda goes like this: I have a service for sale. I want to advertise the service (by buying ad spots on other WoW economic blogs/tools and/or Google adwords). But before I do that, there are things that need to be tightened up. And I think of lots of those things to polish before calling it done.

One example is the Youtube videos. I haven't forgotten them, and I fully intend to complete the series of about 8 videos. I've just been busy coding instead of recording. Making those videos is a chore, let me tell you. I'm running Linux at home, and the software I used for the first videos only uses "open" video formats, which most decent video editors and Youtube don't accept. So I've had video conversion issues (which I may solve by using ffmpeg to record next time instead). Then I have to edit the video to take out all my "ums" and rambly mumbly bits that you don't need to hear. Then I upload it, and write a transcription, so it'll have accurate synced closed captioning. Then I find the "chapter" timestamps to put in the video description. I know I don't need to do all that at the end, and that most users won't care to see the video, never mind the closed captioning, but I want to do it right because I can, and I want to have little out-of-the-way links on the various pages that say, "are you lost? here's a video about this page to give you a tour."

New thought: maybe just use javascript and embedded audio right from the page? Hm.

Anyway, you see how this is going. I want to add something that gives that little extra polish, even though it usually ends up as a lot of work.

Wall of text of stuff done recently, in no particular order:

• The progress bar on items with lots of auctions (think herbs and ore) would linger at that "Getting item price and availability over the past 2 weeks.." step. Now it has accurate(!) tiny little steps as that big step progresses, so you don't stare at a motionless loading bar for too long.


• Gave another look at Yahoo's Best Practices for Speeding Up Your Web Site. I already had the CDN from two weeks ago, and I've implemented other suggestions months before that, but there was more I could do.
  
  
  • In an effort to make fewer HTTP requests, I thought about the page loading process. For items, sellers and categories, when we don't have fresh data in the cache, it shows the progress bar. The progress bar is accurate while loading, and once all the data is collected, it's pushed into the cache, and the page reloads. Then, on the second load, it's a cache hit, and it loads the actual page fairly quickly.
    
    I went through a bunch of effort so, at the end of the data collection, instead of reloading, it hides the progress bar section of the page, and continues with the actual page with the graphs and everything else. This meant no reloads, and fewer HTTP requests. However, it also meant I couldn't use gzip compression anymore (since, to show the progress bar, I have to flush to the browser frequently, which doesn't permit the use of gzip on my server). I ended up quadrupling the bandwidth used in order to save an HTTP request. Boo. That lasted about a day before I switched it back.
  
  
  • I implemented a "wait" in the progress bar display. The idea was, if the page took less than 2 seconds to load, then instead of showing the progress bar, I'll just wait until the page collects all its data and present that. If it takes longer than 2 seconds, then push the progress bar to the browser, and work as it always did. However, that just made the site look laggy (since nothing happened for up to 2 seconds after a link was clicked). That change ended up reverted, as well.
  
  
  • I put all the javascript at the bottom of the page, both in-line scripts and those linked in. The hope was that the page would show something quickly after the browser started getting data, then do all the scripty bits afterwards to fill in the charts. It may have helped, a little. Maybe.
  
  
  • Minimized HTTP requests by putting the 4 login icons in the top left into an imagemap instead of having 4 files. Hey, it's something, and it's on every page.
  
  
  • I used two different domain names for the CDN, to load objects in parallel. I was gonna have a bunch hanging off of tuj.me but then noscript would cough when you tried to load scripts from that new domain. I decided to keep all scripts on theunderminejournal.com and maxcdn.theunderminejournal.com but load most/all static images from cdn.tuj.me. Using the tuj.me domain also means we don't send cookies with every CDN request.
  
  
  • I minified much of the static javascript and some of the dynamic javascript that's in-line in the pages. I tried to minify everything in the HTML that gets sent to the browser through clever use of regexps, but that caused problems and ended up increasing server-side load time anyway.


• When new syndicated editorials are posted here, they're also broadcast on Twitter at @UndermineJrnl with an appropriate pretty small URL. :)


• The above change also forced me to finish the move of RSS collection off of my home server and completely onto the main web server. It had a lot of old code and processes from when I manually approved and edited posts.


• PubSubHubbub! (PuSH) We do a lot with RSS feeds here, and I never looked much into this push technology until this week.
  
  
  • For starters, all the syndicated editorials work via pulling RSS. Just under half of the blogs we watch use PuSH to notify subscribers as soon as new content is available. I wrote a small PuSH client into the feed aggregator to subscribe to feed notifications. That way, for participating blogs, as soon as they post, the article shows up here. No lag. Cool stuff. I saw it work for one post already, but not all blogs participate, and it's been a slow news week besides. I think it works.
  
  
  • Second, I'd like RSS feeds for logged-in users to be more immediate.. i.e., use PuSH as publishers. Technically, we have thousands of feeds, one for each user, but each feed would only have one subscriber, so I didn't want to use a public hub to announce notifications for everyone when nobody might be listening. Instead, I've implemented a custom hub for the site. Since I can keep track of when a feed has a PuSH subscriber, I can announce directly to the subscribers without wondering if the hub has any listeners.
    
    Long story short, this should make RSS feeds update more quickly for users. I've had trouble testing it, though, since it's not 100% immediate between when a reader receives a PuSH notification and the post shows up in the feed reader. It's hard to tell if a new post shows up due to the PuSH note or just from a scheduled check. I think it works. I've been trying it on my own account for the past day or two, and will roll it out to all users in the coming days. Again: no big difference or announcement with this, and the users don't have to do anything different, but it should make things work just a little bit better.
  
  
  • In anticipation for making the main page RSS feeds PuSH "compliant", I've consolidated them to one URL, instead of separate URLs for each realm. I may make the forum RSS feeds publish to a hub, too.. not sure yet.

I also want to explore perhaps having "holiday" header images. Nothing overdone or flashy, because I know there's much to be said for a consistent, clean interface. However, I thought about seeing if I could pull the holiday garlands that you'd see hanging from doorways in-game from the game files, then overlaying those on top of the title graphic during those holiday seasons. Maybe. It might look like crap, Idunno. I just want to see if I can pull the images first, and go from there.

All this talk about PuSH tech on the feeds made me explore XMPP for notifications once again. This could announce the same stuff you get via email/rss to you directly via instant message, if you use Google Talk or any other XMPP-compatible service. Just started looking into this. Not thrilled with the steep learning curve to automatically send messages via an XMPP service, but perhaps there's a decent PHP library out there for this. I'd host my own XMPP server to send the messages for this, natch.

I looked for a long time at potentially supporting SMS notifications. The problem is cost. The cheapest I could send an SMS is $0.01 per text, and even at that price, I'd lose money with the volume of the market notification users. And, what could you do with SMS that isn't already done with GMail? To act upon a text, you'd have to fire up the game, a web browser, or the mobile AH app anyway. GMail instantly pushes emails to your android phone, so if you have email notifications sent to your GMail, you're done, and with more detail than could be squeezed into an SMS. And if someone suggests provider Email-SMS gateways, I'll remind you that each cell provider has a different address, and they can change, and some of them squeeze in the "from" and "subject" lines (which take up room), and some don't, and it's just a mess. So, SMS is out, again. Use email that your smartphone can read, and/or an RSS reader, which will soon should get even more prompt when I turn on the PuSH system.

So I'm getting a bunch of ideas lately, and generally coding like a fiend. And most folks won't notice. But I'm happy, because in the end, the site just keeps getting better and better. :)

So we can't see Guardian Cubs via the API yet. I posted that information on the item page and on the front news page.

Unfortunately, the new Guardian Cub pet is not included in the auction data we get from Blizzard. While the pet may be on your auction house in-game, it is not shown on the WoW Remote Auction House or via the Blizzard API.

Please visit this Battle.net forum thread and reply to let Blizzard know that you want to see the Guardian Cub prices online!

Then I looked at my referrer logs from yesterday, and got a lot of hits from different sites, all talking about the Cub. One of them was this MMO-Champion "how much" thread. Prices were all over the map. Nobody knows what these things are worth. That's understandable, it's a brand new item with a completely new acquisition method.

However, it got me thinking. Maybe Blizzard intentionally left this item out of the API? Maybe they don't want pricing statistics available, because they stand to gain money whenever the gold price is too high for players who'd rather pay cash. If you see a price at 20k for the cub, you may say "screw it" and pay cash. However, if you know that the prices are screwy, and they're trending downwards and you'll probably get one for 10k or less in a few days, maybe you'll wait?

Hard to say. Maybe it's just an innocent "oops we forgot to add that item to the web servers." But maybe not?

edit Heh, and just as I post this, a blue comes in with:

Thank you for the report - its a bug and we're working on getting it resolved today.