blog.humaneguitarist.org » HTML5

jAUs 3-D: no glasses required

nitin — Thu, 12 Jan 2012 00:55:57 +0000

So, I am slightly obsessive compulsive.

I worked a little more tonight on this jAUs thing to add support for start and stop time attributes in the HTML5 tag.

Video should work, too, with a little work, but I don't care about that right now.

What I did tonight was make it so that if a "stopTime" attribute is used, then after that point is reached the player will move the scrubber back to the original "startTime" value though at that point the playback is already paused.

If there is no "stopTime" attribute, then after the file's played itself to the end, the scrubber will move back to the "startTime" value. Again, playback is already paused at that point.

I've tested this with Firefox 9.01, Internet Explorer 9, Safari 4.0.5, Opera 11.60, and Chrome 16.0.912.75 on my Windows 7 (32-bit) laptop.

Running everything from my desktop, I had no problems except that I should mention that if I used the "autoplay" attribute and set it to "true", Chrome didn't start the playback as it should, but it seems that maybe that's a Chrome problem that others are having, too.

Testing with the files uploaded to my hosted account was a different story. Opera seemed to need a page refresh before the scrubber would locate itself at the proper positions – though adding a pesky alert() at the beginning seemed to make Opera happy. Chrome and Safari seemed to take a few seconds to get situated, although they seemed to generally need a restart to move the scrubber to the right place for the last audio player. I didn't test the alert() thing for these two. Firefox did well although I hate the way Firefox moves the audio players around depending on whether the audio has been played or not. And that leaves IE9 … which, hands down, did the best. Maybe that's because of the exception I'd added for it as I mentioned in an earlier post.

So, there's still work to do and things to investigate.

Also, I haven't tested this with really long files or anything so I don't know how that would go. But then again, as I've heard others say, HTML5 media elements aren't really for long-form media anyway.

Oh yeah, one more thing. I did in fact change to "data-startTime" and "data-stopTime" to make the HTML legal HTML5.

Here's the HTML5 code itself, letting one see that the JavaScript has now been moved into a separate JS file.

jAUs

This is the entire recording of Shakespeare's Sonnet 130, read by Nathan Miller for LibriVox.

Your browser does not support the audio element.

"My mistress' eyes are nothing like the sun." - start = 10; end = 13.

Your browser does not support the audio element.

"End of poem."- start = 57, no end specified.

Your browser does not support the audio element.

Oh, and here's the JavaScript file:

/*
***** Note: This software is still in ALPHA. Please refrain from using
the code without first contacting Nitin Arora at nitaro74@gmail.com.
Thanks!
***** 

jAUs: JavaScript  Shark.

Copyright (c) 2012 Nitin Arora. 

Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software. 

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 

jAUs is licensed under the MIT license:

http://www.opensource.org/licenses/mit-license.php.

*/

function jAUs(){

  audioTagArray = document.getElementsByClassName("jAUs");

  for (i=0;i<=audioTagArray.length-1;i++){
    var thisAudioTag = audioTagArray[i];
    jAUs_2(audioTagArray,thisAudioTag);
  }
}

function jAUs_2(audioTagArray,thisAudioTag){
  //if this is placed directly into jAUs() - i.e. not a separate function,
  //then this whole thing doesn't seem to work.

  if (navigator.appName == "Microsoft Internet Explorer"){
    thisAudioTag.onloadeddata = function(){
      var thisAudioTag_startTime = thisAudioTag.getAttribute("data-startTime");
      thisAudioTag.currentTime = thisAudioTag_startTime;
    }
  }

  else {
    var thisAudioTag_startTime = thisAudioTag.getAttribute("data-startTime");
    thisAudioTag.currentTime = thisAudioTag_startTime;
  }

  var thisAudioTag_stopTime = thisAudioTag.getAttribute("data-stopTime");
  var stopString = "jAUs_3(this.currentTime," + thisAudioTag_stopTime + "," + i + ");";
  //returns "jAUs_3(this.currentTime, 13, i);" where "i" is an int.

  thisAudioTag.setAttribute("ontimeupdate",stopString);
}

function jAUs_3(this_currentTime,thisAudioTag_stopTime,i){

  if (thisAudioTag_stopTime){
    //if there's a data-stopTime attribute then ...
    if (this_currentTime > thisAudioTag_stopTime){
      //... reset audio to data-startTime when data-stopTime is reached.
      audioTagArray[i].currentTime = audioTagArray[i].getAttribute("data-startTime");
      audioTagArray[i].pause();
    }
  }
  else if (audioTagArray[i].ended == true){
    //if there's no data-stopTime, move back to data-startTime when playback has ended.
    audioTagArray[i].currentTime = audioTagArray[i].getAttribute("data-startTime");
    audioTagArray[i].pause();
  }
}

window.onload = function(){
  jAUs();
}

Update, January 12, 2012: Turns out "jAUs" is the name for some robotic SDK and I'm not too crazy about that name anyway. So, I'm leaning toward "m(AUj)ulate" (pronounced like 'modulate') which would stand for something like "My Untimely Little Audio Tag Extender". The word "untimely" being, of course, a play on the fact that time is what this is all about. The parenthetical bit refers to "audio" (AU) and JavaScript (j).

And, yes, I care much more about the name/acronym than the script itself.

Update, January 15, 2012: OK, this is interesting. If, for the source of the audio file, I actually use the audio files on the Archive.org site for the LibriVox recordings like so …

  
    
    
    Your browser does not support the audio element.

… then this seems to be working OK in all the browsers. Chrome seems, per casual observation, the slowest in terms of getting the scrubber moved to the appropriate points, but I guess this is progress.

Related to all this stuff I've been messing with, I found this: Consistent event firing with HTML5 video – Dev.Opera. But here, too, they use an alert() to notify the user that metadata is loaded using "onloadedmetadata", but in my tests it seems like the alert() function itself was what was fixing the inability of some browsers to set the current time as my script was instructing.

jAUs: trying to add a startTime attribute to the audio tag

nitin — Mon, 09 Jan 2012 17:29:52 +0000

I work up this morning feeling unwell, but I still went in to work for a few minutes to enable Remote Desktop as it had stopped working (was a Windows Firewall thing) …

Anyway, on the walk back home I started thinking about something I've wanted to play with for a while.

And that's seeing what it would take to add support for a "startTime" attribute for the HTML5 tag using a browser's built-in player. I think it's a real oversight that there isn't native support for passing the start and stop times in the and tags themselves. Um, there isn't right?

Adding support for a "stopTime" attribute would, I think, require more than I'm willing to think about right now (I should be napping) because obviously that entails checking the current time against the point at which one would want the media to stop.

But adding a simple (and, yes, non-standard) "startTime" attribute inside the tag seems pretty easy – if you aren't using IE. Ugh.

As you can see in the code below there's an alert() that returns a null, but without it IE (version 9) won't set the audio players to the values in the "startTime" attributes. The other "major" browsers do fine without it. And, really, it can't be there because it's annoying as hell.

But I think this still points to the problem of each browser having its own player. Take Safari for instance. It doesn't natively show the current or total time of the track. Chrome doesn't show the total time.

Still think Flash is a bad thing?



  
    jAUs
    
  


  
    
    
    Your browser does not support the audio element.
  

  


  
    
    
    Your browser does not support the audio element.

ps: If this ever turns into a little project of mine, it needs a decently cool name to keep my interest. "JS" for JavaScript and "AU" for audio mashed up equals "jAUs". I hope that isn't already taken within the JavaScript/HTML5 domain.

Update, January 09, 2012: Now that an hour has past and I finished watching "Thor & Loki: Blood Brothers", I figured out how to elegantly do the "stopTime" thing – well, at least I think it's better than what I was thinking earlier today!

So unlike in the code above, the JavaScript below only hits on one element (the one with an "id" value of "jAUs") and not all the ones with a "class" value of "jAUs". But the point is that this new code adds the "ontimeupdate" attribute to the tag and makes it run a function called stopper() that will stop the audio once the current time is greater than the designated stopping time.

Now, there's still work to do. For example, the code should test for the existence of the "stopTime" attribute before forcing the stopper() function to run, but that's no big deal. I also need to test doing this as above – i.e. hitting all tags with a "class" value of "jAUs" – and see how that works. And there's that pesky IE thing, too.

Anyway, this is working in the other "major" browsers 'far as I can tell.



  
    jAUs
    
  
  

  
    
    
    Your browser does not support the audio element.

Update, January 09, 2012: I still don't like it, but it seems that the alert() line can go at the top of the jAUs() function and still allow all this to work on IE 9. At least that prevents an alert message for each tag within the page.

Update, January 10, 2012: Ah. Now, we might be getting somewhere. I don't know if it's just a band-aid fix, but if I add an "onloadeddata" bit for IE9 this actually seems to be working just fine. The problem was that IE didn't consider the element's properties to be accessible, thus giving me an "Invalid State Error" with a code number of 11 which is this, more or less: "An attempt was made to use an object that is not, or is no longer, usable." So, IE9 is taking a little longer than the other browsers in realizing what's what.



  
    jAUs
    
  
  

  
    
    
    Your browser does not support the audio element.

Update, January 10, 2012: Hopefully (for you and me) this is the last update for a while. Hey, I'm still sick, jerk!

Anyway, this is a test for hitting on all tags with a class value of "jAUs".



  
    jAUs
    
  
  

  
    
    
    Your browser does not support the audio element.
  

  


  
    
    
    Your browser does not support the audio element.

By the way, using the HTML5 "data-" prefix a la "data-startTime" would make the attribute(s) valid, though still not part of a standard per the tag specifically. But I guess the "data-" prefix is an acknowledgment of reality.

SAVS: a Simple Audio/Video Synchronizer

nitin — Mon, 05 Sep 2011 13:39:59 +0000

About a year ago I did some text to audio synchronization tests with HTML5 and Flash.

The tests were partially successful, but I think what really mattered is that I set four goals that I felt needed to be met before the word "synchronization" could truly be used:

The user should be able to click on a line of text and hear the related media.
The user should be able to "scrub" ahead on the media player and the text should follow.
The page should report where in the document the user is.
The page should automatically keep the media/text synchronized without user intervention.

Basically, I've seen a few people make it so that you could watch media while the transcript text was also on the page (scrollable as opposed to overlaid closed captions) and the user could click on a line and have the movie/audio skip ahead to that moment (goal #1). That's great and all, but that's not synchronization.

Synchronization is a two way street and I've been working this past week during what I'm calling "4 days of madness" to come up with a really simple solution to real synchronization. I did run across this really cool RadioLab page that achieves goal #1, but as much as I like it I want more features with less flash (as in "flash and dash" not Adobe Flash!) and less code. No mistake: it looks fantastic and I also appreciate that they've got the text timed to clusters of a couple of words rather than by line but the only thing I've seen that gets it all "right" per my perspective was a subscription resource by Alexander St. Press. It achieved all the goals above using a Flash player and the rest appeared to by done with Javascript and some jQuery smooth scrolling. It was also timed by clusters of words and not just by line or by paragraph. Of course, conceptually it's the same whether one marks up their text – in the temporal sense – by line or by word, but it's a little more work to do it by word of course. Unfortunately, I've seen people do the opposite: they use a static unit of time like 60 seconds and only mark up the text every minute. That's taking the easy way out and also misses the point entirely since it makes the text subservient to an arbitrary unit of time. Would it be acceptable if closed captioning and subtitles on your foreign films only showed up in large chunks every minute? I would hope not, and in the case of the former it would violate the the spirit if not the letter of the "law" in regard to accessibility. If done right, you can use the same timed text file to both serve up captions in addition to showing the full text on the page. It's more time and cost efficient to re-purpose the same data for two needs.

Anyway, let's get back to Alexander St. Press. I loved what I saw when my boss (I work at NC Live) showed it to me. I got really excited and said something like, "This is what I've been waiting to see!". In addition to the great and true syncing, they also had a feature that would let the user make and share clips, much the way you can on sites like NBC's Meet The Press. The Alexander St. Press site also allowed you to annotate that clip, which is a great feature for teachers and librarians, etc. Alexander St. Press also has this with their classical music streaming subscription service, which in the spirit of full disclosure I pay for. They ALSO had a cool timeline where you could see what I call "hot spots" – places where others had made clips. The idea, I guess, is that spots on the timeline with more clusters would indicate a particular point of interest. Nothing new, because you see that all the time with streaming sports like the US Open's site where you can go back and watch previous moments in matches and then "go live" at any time. But the difference is, of course, that Alexander St. Press was using user-contributed clips.

So long story short (or just not as long), in a few weeks I need to present these ideas to some people and talk about how we think these features could be useful for our users. And the more I struggled with how to talk about these concepts without a prototype the more I thought I would a) sound like I'm crazy and b) like I'm full of hot air.

I decided that it was time to go back to some earlier tests of mine from early April and just build a prototype so we could just show it to people and not have to talk theoretical speak. I think it's generally easier to explain and convince people of the utility of software by showing it rather than telling it. Actions > words, right?

Well, early tests are working and only required me to add one line of Actionscript to our current Flash player and about only 50 lines of Javascript code are needed to keep the text and media synced. The tests I did were for some PBS videos we purchased along with closed captioning files.

I was so excited that it was finally working that I went home during those "4 days of madness" to write an HTML5 version which is virtually identical to the Flash version. It's got basic clip making features as well as a very basic tool inspired by this video score tutorial to make timed text files provided you have the audio and full text in hand. Eventually, I'll comment the code up and improve some options and post a download to the source for the HTML5 version. At work, we'll probably eventually offer the code as it's tweaked to meet our aesthetic needs, etc. As you'll see in the demo video below, I have no aesthetics!

I'll shut up now and leave you to the video if you're interested. I recommend watching it in HD so you can read the words on the page.

As my friend whom the HTML5 version is kinda named after likes to say:

More later …

SAVS: a Simple Audio/Verse Synchronizer from nitin arora on Vimeo.

…

Update, September 20, 2011: To avoid confusion as to what this does, I'm renaming this from "Simple Audio/Video Synchronizer" to "Simple Audio/Verse Synchronizer" or something …

Update, October 16, 2011: Cool, I found one more thing that meets all the four goals at http://www.dinglabs.com. They're pitching it as a foreign language learning tool, but same difference. Also, that site led me to TranscriberAG, a tool for transcribing audio.

segmenting audio with AudioRegent, SoX and XML

nitin — Sat, 16 Jan 2010 19:05:39 +0000

For some reason I feel obligated to point out that I haven’t blogged in a while for a few reasons:

Christmas break from school/work at the University of Alabama
the desire not to blog for the sake of blogging
and …

I’ve been working on something huge – at least for me. It’s a piece of software called AudioRegent that harnesses XML to create derivative "clips" of regions within WAV audio files. A region is simply a user-defined segment within an audio file, like a track on a Compact Disc.

Besides writing the program in Python, which I pretty much finished in December, I had to also develop the XML format which I call SimpleADL (Simple Audio Decision List) that AudioRegent looks at and then makes derivative audio clips by leveraging SoX, the Sound Exchange command line audio editor. AudioRegent and SimpleADL can also be used to sync audio to text, like transcripts.

Actually, the programming and devising SimpleADL were the easy part. The hard stuff was the documentation and deciding on a license for the software.

I tried to find a balance in documenting the software: being thorough without writing a novel. I’m not sure I succeeded, but I can always improve it with time.

I used the W3C’s Amaya editor to write the documentation in XHTML. Sure, you can use OpenOffice to export a document to XHTML, but man is it bloated and messy. Amaya writes really clean XHTML.

As for the license, I chose the BSD license. As I understand it, this allows one to use the source code at will in future open or closed-source applications as long as you maintain the credits for AudioRegent. I was tempted to use the Mozilla Public License (MPL) which, again from what I can tell, is similar to the BSD license except that any source derived from AudioRegent would have to stay open-source though any peripheral code can be closed-source. I absolutely decided against the GNU General Public License which is viral and imposes its philosophy perpetually on all subsequent code, even peripheral code. Some have even argued that it works against its own objectives and is less "open" than the MPL.

Now I realize that, practically speaking, a skilled programmer could write better code from scratch in 30 minutes as opposed to the some 30 hours I needed, but I wanted to go about this quasi-professionally. And I learned more about licensing, which was cool.

Anyway, rather than try and explain the software itself and how to get it, I’d be better off pointing you to the documentation if you have any interest …