//Saurabh ShrivastavaJekyll2018-08-12T23:42:27+05:30https://saurabhshri.github.io/Saurabh Shrivastavahttps://saurabhshri.github.io/saurabh.shrivastava54@gmail.comhttps://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-week-9-10-let-s-zoom-in2017-08-23T00:00:00+05:302017-08-23T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Almost two weeks have passed since the second evaluations and my last blog post. This blog post lays down the summary of work I’ve done in past two weeks.</p>
<blockquote>
<p>The task checklist with deliverables is located here : <a href="https://saurabhshri.github.io/gsoc/">https://saurabhshri.github.io/gsoc/</a> .</p>
</blockquote>
<p>The major work during these two weeks were to implement phoneme recognition. This involved generating phonetic language model with the help of corpus. The challenging task here was to convert text corpus into phonetic corpus.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><s> princeton </s>
<s> a fine institution </s>
IL P R IH N S IY T AH N SIL
SIL AH F AY N IH N S T AY T UW SH AH N SIL
</code></pre></div></div>
<p>While creating dictionary, I am using seq2seq to generate the phonemes from words which is trained on CMUDict. But this process is extremely slow and prooved not fesiable for large corpus. So, the alternative was to use fixed sets of rules to perform the conversion. Of course this isn’t going to be as accurate as the one generated from seq2seq, but since this corpus’s primary aim is to generate language model, it was the way to go.</p>
<p>Thankfully, these rules were compiled into C++ rules by <a href="github.com/DanielSWolf/">Daniel S. Wolf</a> in his project <a href="github.com/DanielSWolf/rhubarb-lip-sync">Rhubarb Lip Sync</a>, and I modified it’s G2P file to perfrom corpus to phonetic corpus conversion.</p>
<p>On the left are the result of the tool that uses tensorflow to learn the rules using a dictionary (very slow) and on the right are the results obtained using the script I made which follows the rules I found above.</p>
<blockquote>
<table>
<tbody>
<tr>
<td>america</td>
<td>AH M EH R AH K AH</td>
<td>AE M EY R AY K AE</td>
</tr>
<tr>
<td>hypocrit</td>
<td>HH IH P IH K R IH T</td>
<td>HH AY P AA K R IH T</td>
</tr>
<tr>
<td>trump</td>
<td>T R AH M P</td>
<td>T R AH M P</td>
</tr>
<tr>
<td>saurabh</td>
<td>S AO R AE B</td>
<td>S OW R AE B HH</td>
</tr>
</tbody>
</table>
</blockquote>
<p>This corpus is then used to generate phonetic language model which is fed to phonetic decoder to perform phoneme recognition. Performing phoneme recognition slows down the process relatively as it adds an extra cost to compute those terms.</p>
<p>I also encountered an interesting bug during this. If the corpus contained dual whitespaces, the language model generated was in corrupt form. But the error was not properly reported by SphinxBase and upon reporting, a fix was pushed for the same.</p>
<p>I then proceeded to add deep level logging throughout the program which should aid in debugging. The logging is implemented through a variadic function which is invoked through a defined macro. Similarly, I implemented a function and macro for handling cases for fatal cases with defined error codes. These can be found in <code class="highlighter-rouge">/src/lib_ccaligner/commons.h</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 249 | Wave File chunkID verification successful
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 253 | Begin decoding wave file
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 108 | File format is identified as WAV
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 115 | Finding FMT and DATA subchunks
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 130 | FMT index : 12 , DATA index :70
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 153 | PCM : True
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 162 | MONO : True
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 171 | Sample Rate 16KHz : True
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 184 | BitRate 16 bits/sec : True
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 202 | Number of samples : 34543104
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 203 | Reading samples
[INFO] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/read_wav_file.cpp : 257 | File decoded successfully
</code></pre></div></div>
<p>and</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ERROR] /Users/saurabhshri/Desktop/try/ccaligner/src/lib_ccaligner/params.cpp : 127
-oFormat requires a valid output format!
</code></pre></div></div>
<p>In the following weeks I’ll implement the unified output handler for all possible situtations - continuous mode, complete mode or transcribing mode. I’ll also remove deprecated code, organise and refactor the remaining code while completing the documentation. Memory leak fixes, further optimisation et cetera shall also be done next week.</p>
<p>Hoping everyone’s having fun! See you in the next one! 🙏🏻</p>
<p><a href="https://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-week-9-10-let-s-zoom-in">Google Summer of Code, Week 9 & 10 : Let's Zoom In! </a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on August 23, 2017.</p>https://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-the-mid-term-evaluations2017-08-22T00:00:00+05:302017-08-22T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>With the ending of eight week of Google Summer of Code, second phase of conding is almost complete. This meant that it’s time for mid term evaluations, which would judge whether I will be continuing the program or fail.</p>
<blockquote>
<p>I successfuly passed my mid term evaluations, thankfully, leaving my mentors happy with my work. Though initially I hit a bump, but later on, I passed with flying colours. 😊</p>
</blockquote>
<p>This was the last week of second phase of coding period. In the previous 4 weeks I began full blown ASR work and integrated it with the work built during the first phase. That involved couple of things like using PocketSphinx to perform speech recognition, generating language model, grammars, dictionaries et cetera from the subtitles followed by actually performing alignement using fuzzy comparision and window based search. The fourth week was reserved as buffer week to meet the missed milestones, complete the documentation and all the remaining things. By the end of second phase, I met the following deliverables :</p>
<ul class="task-list">
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Word recognition and timed transcription.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Tuned language models and dictionaries.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Adaptation script and implementation for custom models and dictionaries based on subtitles.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Exporting result with colored identification of recognised/ non recognised words. (Just for demonstration purpose)</li>
</ul>
<p>When we began evaluation, with the first sample, output of ccaligner was mostly rubbish with no relevance to the actual subtitles. The sample was from one the daily soap operas. Turns out that the subtitle etracted from that vides were roll-up subtitles and had delayed timing, resulting in different recognition then expected, as the expected format is a SubRip.</p>
<p>Next we tried another sample from the same genere, and this time, the program kind of froze. It kept on trying to understand the utterance, but kept failing. After debugging, I found out that the subtitle had some emojis in it, which wern’t being handled correctly. This resulted in corrupted grammar and language model leading to the freeze. It was a rather quick fix, but Carlos ultimately decide to shift the <em>actual</em> evaluation later, so that I can do a bit more testing. Wise decision, as I found out that when the audio had no speech as opposed to the indictaion present in subtitle and flase positive from VAD, the decoder crashed.</p>
<p>After handling non ASCII characters and fixing the crash, I added the transcribe option in which the decoder does the alignment irrespective of subtitle timings. It still uses subtitle file to create language model, but marks the utterances on it’s own and gives word by word timed transcription along with confidence score.</p>
<p>Two days later, at night, I found out that Carlos is free and we <em>re-attempted</em> the evaluations. This time it worked flawlessly and I think I met all his expectations! 😊 He was happy with the output, and
he proceeded to tell me that I passed, and said, that I did a good job! I was very happy to hear this after hitting that bump before, and kind of not meeting his expectations in the first evaluations.</p>
<p>Few days later I received the official email from Google Open Source Office, declaring me passed in the mid term evaluations. In the official feedback he mentioned about completing documentation as well and giving credits everywhere (which, I thankfully did throughout)!</p>
<blockquote>
<p>Looks like the hardest parts are done and working reasonably well. Important missing thing is documentation (good one, both technical and informative). Your blogs are well written though so I don’t think this is going to be a problem. When you document, remember to mention all dependencies, too. Also if code was taken from any other project, give credit.</p>
</blockquote>
<p>Here’s the screenshot of the same :</p>
<p><img src="/images/posts/mid_term_evaluation_gsoc2017.png" alt="mid_term_evaluation_gsoc2017.png" /></p>
<p>I’ll continue to improve myself and learn more, as I have learned from this evaluation.</p>
<p>Thank you Carlos and Alex for being my mentor, and for passing me in the second evaluations! 😊</p>
<h3 id="whats-next">What’s next?</h3>
<p>Now I will begin working on implementing phoneme recognition in the tool. This involves using phoneme decoder, generating phonetic language model and implementing it in CCAligner. This will be followed by all sorts of refining and handling output. Logging, error handling, external documentation shall be of utmost importance as well. The next month will be super busy as it’s high time in college too.</p>
<p>I hope other students have passed their evaluations as well and are enjoying working. I can not express how amazing the experience has been so far. It’s almost surreal. Hope to complete the project within the timeline. See you in the next one! 🍀</p>
<p><a href="https://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-the-mid-term-evaluations">Google Summer of Code : Mid Term Evaluations!✍️ </a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on August 22, 2017.</p>https://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-week-7-8-let-s-karaoke2017-08-06T00:00:00+05:302017-08-06T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>This blog post covers the work done in the last two weeks of the second phase of coding. I have updated the checklist of the deliverable for the second phase at <a href="https://saurabhshri.github.io/gsoc/">https://saurabhshri.github.io/gsoc/</a> .</p>
<blockquote>
<p>If you are wondering about the delay - I am not being lazy, blogging is one of the things I love the most. I am just a little busier than ever. 🙂</p>
</blockquote>
<p>In the <a href="https://saurabhshri.github.io/2017/07/gsoc/google-summer-of-code-week-5-6-what-d-you-say">last blog post</a> I mentioned that one of the tasks of this period would be to figuring out a proper way of handling FSGs so that they consider the garbage loop as well. So, after a lot of experimenting with different scenarios and different weights, I finally figured out a good enough approach with a consistent efficiency irrespective of the audio. Creating FSGs dynamically should not be considered as the most accurate approach as devising them manually based on audio could substantially increase the accuracy.</p>
<p>Here’s an example of one such FSG :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FSG_BEGIN CUSTOM_FSG
NUM_STATES 20
START_STATE 0
FINAL_STATE 19
# Transitions
TRANSITION 0 1 0.0909
TRANSITION 0 2 0.0909
TRANSITION 0 3 0.0909
TRANSITION 0 4 0.0909
TRANSITION 0 5 0.0909
TRANSITION 0 6 0.0909
TRANSITION 0 7 0.0909
TRANSITION 0 8 0.0909
TRANSITION 0 9 0.0909
TRANSITION 1 10 1.0 i
TRANSITION 2 11 1.0 was
TRANSITION 3 12 1.0 offered
TRANSITION 4 13 1.0 a
TRANSITION 5 14 1.0 summer
TRANSITION 6 15 1.0 research
TRANSITION 7 16 1.0 fellowship
TRANSITION 8 17 1.0 at
TRANSITION 9 18 1.0 princeton
TRANSITION 10 19 0.0909
TRANSITION 11 19 0.0909
TRANSITION 12 19 0.0909
TRANSITION 13 19 0.0909
TRANSITION 14 19 0.0909
TRANSITION 15 19 0.0909
TRANSITION 16 19 0.0909
TRANSITION 17 19 0.0909
TRANSITION 18 19 0.0909
TRANSITION 19 0 0.0909
FSG_END
</code></pre></div></div>
<p>This makes the grammar flexible and makes a room for allowing the cases where recognition doesn’t match the expected output!</p>
<p>The next part was to actually align the recognised words with the audio. Prior to this, the recognition and time detection based on frames was working. The very first step in this was to reset the time stream, so that the frame count can be with respect to zero. This eliminated the chances of error as we are not processing all samples, but only those samples which have utterances.</p>
<p>For each word, I found out the exact timing by dividing the frame count with the frame rate and converting it into milliseconds. This value was then added to the beginning timestamp of the samples for which they were recognised.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>//the time when utterance was marked, the times are w.r.t. to this
long int startTime = sub->getStartTime();
long int endTime = startTime;
/*
* Finding start time and end time of each word.
*
* 1 sec = 1000 ms, thus time in second = 1000 / frame rate.
*
*/
startTime += sf * 1000 / frame_rate;
endTime += ef * 1000 / frame_rate;
</code></pre></div></div>
<p>They are also stored in an object of class <code class="highlighter-rouge">recognisedBlock</code> for later uses.</p>
<p>For the alignment with subtitle, the recognised words needed to be matched with subtitle text. A simple linear search would not be a preferred choice in this case, as it will provide erroneous results. For example, consider the case :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Actual : [Why] would you use tomato just why
Recognised : would you use tomato just [why]
</code></pre></div></div>
<p>So, if we search whole recognised sentence for actual words one by one, then Why[1] of Actual will get associated with with why[7] of recognised. This will not only result in incorrect tagging of word, but also the words in square bracket matches, sets the <code class="highlighter-rouge">lastWordFoundAtIndex</code> at 6, and the search stops.</p>
<p>To prevent this I am using a window based search approach where a word is searched only in the defined window thus, limiting the number of words it can look ahead, in order to prevent error of mismatching. This also enables user to define a window to look into.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>searchWindowSize = 3;
Recognised : so have you can you've brought seven
|
---------------
| |
Actual : I think you've brought with you
Recognised : so have you can you've brought seven
|
-------------------
| |
Actual : I think you've brought with you
</code></pre></div></div>
<p>But since the recognition is not perfect, I perform a fuzzy search to find the match instead of directly comparing the words. The fuzzy search is performed by calculating the <a href="https://en.wikipedia.org/wiki/Levenshtein_distance">Levenshtein distance</a> between the words. I then use this distance to find out how much are the words similar. If the words have similarity of 75% or above, I consider them a match! This percentage of course is user configurable.</p>
<p><img src="/images/posts/aligning_recognised_words.png" alt="aligning_recognised_words.png" /></p>
<p>Once the match is found, the word is marked as <em>recognised</em> and it’s starting and ending timestamps are initialised.</p>
<p>There are various output options added in which the results can be visualised. One of those options is addition of output as Karaoke mode. In this, the words that is being spoken is written in SRT format with a <code class="highlighter-rouge"><font></code> tag. So, when the word is being spoken,it gets highlighted. Here’s a gif with the excerpt from the respective karaoke subtitle file. This subtitle file was the result of karaoke output from ccaligner.</p>
<p><img src="/images/karaoke.gif" alt="Output Visualised as in Karaoke format!" /><br />
<em>Output Visualised as in Karaoke format <code class="highlighter-rouge">--print-as-karaoke yes</code></em> .</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>00:00:12,780 --> 00:00:12,911
<font color='#A1E4D3'> I</font> was offered a summer research fellowship at Princeton
00:00:12,810 --> 00:00:13,100
I <font color='#0000FF'> was</font> offered a summer research fellowship at Princeton
00:00:13,180 --> 00:00:13,540
I was <font color='#0000FF'> offered</font> a summer research fellowship at Princeton
00:00:13,550 --> 00:00:13,900
I was offered <font color='#0000FF'> a</font> summer research fellowship at Princeton
00:00:13,910 --> 00:00:14,290
I was offered a <font color='#0000FF'> summer</font> research fellowship at Princeton
00:00:14,300 --> 00:00:14,680
I was offered a summer <font color='#0000FF'> research</font> fellowship at Princeton
00:00:14,690 --> 00:00:15,130
I was offered a summer research <font color='#0000FF'> fellowship</font> at Princeton
00:00:15,140 --> 00:00:15,250
I was offered a summer research fellowship <font color='#0000FF'> at</font> Princeton
00:00:15,260 --> 00:00:15,940
I was offered a summer research fellowship at <font color='#0000FF'> Princeton</font>
</code></pre></div></div>
<p>Then I worked on creating helpful demonstration for my evaluation. In my first evaluation, proper demo were missing and it impacted my evaluation. So, I made sure, it was not the case this time. This involved creating a simple user interface so that arguments could be issued to check the developed functionalities so far. Of course this UI was just for testing and was not very polished. I also created a simple batch script for dependency installation. If you want to try out the work so far, you may find detailed installation usages in the readme file in the <a href="https://github.com/saurabhshri/ccaligner/tree/development">repository</a>. As always, the recent commits are in <code class="highlighter-rouge">development</code> branch. To try, simply do</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./ccaligner --print-aligned color -in input.wav -srt input.srt
</code></pre></div></div>
<p>I would like to apologise for delayed posts and documentations. I am already in the college as I can not afford to miss it more. I am trying to balance all the aspects of the project, college and placement and I hope to come on track soon! The next post shall cover my mid term evaluation and the changes I made <em>during</em> it. It pretty interesting and climatic I would say. 😉 I hope other participants are having fun, and are doing good! See you in the next one! 🖖🏻</p>
<p><a href="https://saurabhshri.github.io/2017/08/gsoc/google-summer-of-code-week-7-8-let-s-karaoke">Google Summer of Code, Week 7 & 8 : Let's Karaoke! 🎤 </a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on August 06, 2017.</p>https://saurabhshri.github.io/2017/07/gsoc/google-summer-of-code-week-5-6-what-d-you-say2017-07-12T00:00:00+05:302017-07-12T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Almost two weeks have passed since the first evaluations and my last blog post. This blog post lays down the summary of work I’ve done in past two weeks, and is the first time I am writing a combined blog post of more than one week.</p>
<blockquote>
<p>If you’re wondering why I haven’t updated the checklist till now, the answer is simple - I am discovering things on the go. Determining and forcing rigid checklists would only make my work restrictive. I will continue uploading the blog posts with the updates as always. I have major deliverables in my mind as per the proposal, and will be listed in the checklist before evluations. If you are interested in reading my project proposal, it could be found <a href="https://github.com/saurabhshri/saurabhshri.github.io/blob/master/GSoC/5565268630700032_1490805743_Word_by_Word_Subtitle_Sync_by_Saurabh_Shrivastava_CCExtractor.pdf">here</a>.</p>
</blockquote>
<p>I started working on implementing ASR (Automatic Speech Recognition) in the project as soon as <a href="https://saurabhshri.github.io/2017/07/gsoc/news/google-summer-of-code-week-4-the-evaluations">I passed my first evaluations</a>. In the phase 1 of coding period, I built the foundation of the tool, upon which the core alignment has to be built. I also made an <a href="https://www.youtube.com/watch?v=km1iHe_mGuo">approx aligner</a> to aid in the task. By the end of the first phase, the project has met following milestones :</p>
<ul class="task-list">
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Tool for subtitle processing and basic testing architecture.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Sample repository.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Algorithmic and Probability based word - audio matching.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />Audio processing.</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />VAD implementation.</li>
</ul>
<p>In the past two weeks, I have started the work of analysing audio to recognize words. This involves using ASR. I am currently using <a href="https://github.com/cmusphinx/pocketsphinx">CMU’s PocketSphinx</a> for this task. Maybe in the future I’ll add more ASRs like Kaldi or Google Speech Recognition et cetera. PocketSphinx is a (<em>relatively</em>) lightweight, speech recognition engine, specifically tuned for handheld and mobile devices, though it works equally well on the desktop. It’s written natively in C and has wrappers in huge number of languages such as Python or even JS.</p>
<p>Since, the project needs to be - <em>not just an academic tool</em> , the trade off needs to be among resource requirement and accuracy. PocketSphinx kind of stand in the middle of the niche. Plus, thanks to <a href="https://github.com/nshmyrev">Nickolay V. Shmyrev’s</a> for his tireless work at answering questions on forums and groups, the help is few messages away.</p>
<p>The very first step was to get PocketSphinx compile using CMake, since that is what I was using to build the tool. This also meant ensuring that it compiles across all platforms. The tutorial recommends installing PocketSphinx but there was no tutorial for compilation to be used as API. After lots of ‘trial and error’, I ended up collecting all the <code class="highlighter-rouge">.c</code> by recursively going through both PocketSphinx and SphinxBase libraries and then creating object files using them. Since, I am not compiling them the ‘recommended way’, there are certain flags, that remain unset. So, I had to manually set them. If you ever need to compile PocketSphinx in the same way, you may look at the CMakeLists.txt file in the main directory the project. If you have some better alternatives, PR are most certainly welcomed! 😊</p>
<p>After getting it to compile, I tried incorporating it with existing audio pipeline, and it works just fine. I first tried to use it directly on the approx locations, but soon realised that all it yielded me was garbage. It makes sense because we need to mark that starting and ending of “utterances” properly in order for successful recognition. It would need more than just taking some arbitrary window and expecting it to work.</p>
<p>Upon processing one sub at a time, I found out that the accuracy was very bad, it indeed recognizes some words, but for rest, it was out of bounds.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Recognised : an exalt are so i'm
Actual : So, in the next half hour or so,
</code></pre></div></div>
<p>This was expected as it does not make use of the transcription we already have.</p>
<p>So, the next step was to generate a custom language model based on available transcription. The ngram model is generated using CMUCLMTK toolkit. I generate the dictionary containing phonemes using the g2p-seq2seq tool. Currently they are invoked using system commands. I have also started using a new, better acoustic model which has significantly increased the accuracy of the recognition.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Recognised : exploring your vision for one exciting future might look like
Actual : exploring your vision for what an exciting future might look like
Recognised : so this is makes the first question a little ironic
Actual : which I guess makes the first question a little ironic
</code></pre></div></div>
<p>So, basically, I process chunks of samples based on timings present in subtitles. I mark utterance using the beginning time of subtitle and process samples that fall in the duration of the dialogue. I mark end of utterance and find hypothesis. Then I find the times of the words using the segment iterator. In case you’re looking to implement the same, the function looks something like this :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bool Aligner::printWordTimes(cmd_ln_t *config, ps_decoder_t *ps)
{
ps_start_stream(ps);
int frame_rate = cmd_ln_int32_r(config, "-frate");
ps_seg_t *iter = ps_seg_iter(ps);
while (iter != NULL) {
int32 sf, ef, pprob;
float conf;
ps_seg_frames(iter, &sf, &ef);
pprob = ps_seg_prob(iter, NULL, NULL, NULL);
conf = logmath_exp(ps_get_logmath(ps), pprob);
printf(">>> %s \t %.3f \t %.3f\n", ps_seg_word(iter), ((float)sf / frame_rate),
((float) ef / frame_rate));
iter = ps_seg_next(iter);
}
}
</code></pre></div></div>
<p>So far so good. :) This works with good accuracy, but not very good.</p>
<p>I really thought having subtitles will make the work easier as we already have some time information and order in which the words are spoken.</p>
<p>I tried creating FSG as :</p>
<p>If subtitle was :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>00:00:04,960 --> 00:00:06,536
Thanks for having me.
</code></pre></div></div>
<p>Resultant FSG file was</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FSG_BEGIN SAURABH
NUM_STATES 5
START_STATE 0
FINAL_STATE 4
# Transitions
TRANSITION 0 1 1.0 thanks
TRANSITION 1 2 1.0 for
TRANSITION 2 3 1.0 having
TRANSITION 3 4 1.0 me
FSG_END
</code></pre></div></div>
<p>In few cases it helps recognize the words <em>perfectly</em>, but this makes it too restrictive. For the parts where the recognition is something else or there were some other utterance, I get</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR: "fsg_search.c", line 940: Final result does not match the grammar in frame 44
</code></pre></div></div>
<p>I am hoping to make it flexible so that it still knows that the next word to recognize is ‘X’, and it becomes easier for it, but at the same time not fail.</p>
<p>Nikolay, suggested to include the garbage loop in the FSG to make it a little flexible. He was very kind to draw me a state diagram depicting the hints to what to do.</p>
<p><img src="/images/posts/garbage_loop_fsg.png" alt="garbage_loop_fsg.png" /></p>
<p>The next task will be to convert this into FSG programatically for all subs. Also, now since I have the timestamps of the words being spoken, it’s time to do the alignment. The difficult part will be for the words which are not recognised, and based on the type of speech in audio, sometimes they can be large. One way is to assign approx timestamps where the words couldn’t be recognised. I’ll let user enable/disable this with an argument.</p>
<p>On the personal front, my college is resuming quite earlier than I anticipated. Let’s see how it goes. I am still trying to figure out, if I should join right now or not. Since I am stepping into final year, the pressure for attendance should be low (as is usually the case). Let us see how it goes. I hope everything goes fine, because managing both college and full time GSoC project won’t come easy. In worst case, I might have to join college soon. In that scenario, I’ll try to cover-up by working extra on weekends. Also, I need to stress less probably. It’s been a long time I haven’t watched any movie, and ever since Silicon Valley’s season ended, my Monday entertainment ended as well. I loved the new spiderman in Captain America : Civil War, and I hope I get to able to watch the third reboot : Spiderman Homecoming soon. I have to do shopping as well for my college. Since, it’s placement season this semester, I will need formal clothes. I honestly don’t like shopping, specially if it’s related to clothes. I am already nervous and tensed about placements and all these formalities do not help at all.</p>
<p>Do you guys have any suggestions or tips? Feel free to comment / message me about it! 🙂 I hope other participants are having fun working on their projects. I can feel the stress sometimes when things don’t work as I planned them. I am really lucky to be in such an encouraging community who always somehow is able to cheer me and motivate me. Hopefully, everything shall work out good! 🕊</p>
<p><a href="https://saurabhshri.github.io/2017/07/gsoc/google-summer-of-code-week-5-6-what-d-you-say">Google Summer of Code, Week 5 & 6 : What'd You Say?👂</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on July 12, 2017.</p>https://saurabhshri.github.io/2017/07/gsoc/news/google-summer-of-code-week-4-the-evaluations2017-07-03T00:00:00+05:302017-07-03T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>The fourth week ended on 29th of June marking the end of Phase 1 of coding period. The week also comprised of the first evaluations, which would judge whether I will be continuing the program or fail. 💀</p>
<blockquote>
<p>I passed my evaluations! My mentor already told me about it during the evaluations, I received the official email from GSoC team on 30th declaring that I am eligible to continue through the next rounds.</p>
</blockquote>
<p>This was the last week of first phase of coding period. In the previous 3 weeks I spent time building a the nitty-gritty of the tool upon which now I will continue the work. That involved <a href="https://saurabhshri.github.io/2017/05/gsoc/creating-a-full-blown-srt-subtitle-parser">building a robust subtitle parser</a>, <a href="https://www.youtube.com/watch?v=km1iHe_mGuo">creating approx aligner</a>, building test environment (which included collecting samples), <a href="https://github.com/saurabhshri/CCAligner/blob/master/src/lib_ccaligner/read_wav_file.cpp">ability to read and process audio</a>, <a href="https://github.com/saurabhshri/CCAligner/tree/master/demo/VAD">voice activity detection</a> and a lot of reading. The fourth week was reserved as buffer week to meet the missed milestones, complete the documentation and all the remaining things.</p>
<p>In the fourth week I added the capability to read wave files from stream/pipe. Till now the files that were present on the disk were read. Now it is possible to simply pipe the wave file into the program. It was comparatively challenging as the specifications needed to be decoded on the go. It would have been easy if I were only reading raw samples, but reading the wave file and verifying to make sure the wave file is of proper format needed a lot of precision.</p>
<p>There are 3 modes in which wave files can be read.</p>
<ul>
<li>File is present on disk. This is the most common usage scenario when the file is present in the disk. If filename is passed to <code class="highlighter-rouge">WaveFileReader</code> constructor then it uses this method.</li>
</ul>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> WaveFileData * file = new WaveFileData(argv[1]); //supply filename
file->read();
std::vector<int16_t> samples = file->getSamples(); //return samples
</code></pre></div></div>
<p>For example : <code class="highlighter-rouge">./ccaligner input.wav</code></p>
<ul>
<li>Data is piped/streamed. This is helpful when the wave file is not present butis being generated. This helps in making the tool capable of fitting into pipelines.</li>
</ul>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> WaveFileData * file = new WaveFileData(); //will read from pipe or stream
file->read();
std::vector<int16_t> samples = file->getSamples(); //return samples
</code></pre></div></div>
<p>For example : <code class="highlighter-rouge">ffmpeg [arguments] | ./ccaligner</code></p>
<ul>
<li>Data is piped/streamed and is first stored in buffer and then processed. This is helpful when we need to ensure that we have complete data before proceeding. This too helps in making the tool capable of fitting into pipelines.</li>
</ul>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> //readStreamIntoBuffer is an enum decalared in read_wave_file.h
WaveFileData * file = new WaveFileData(readStreamIntoBuffer);
file->read();
std::vector<int16_t> samples = file->getSamples(); //return samples
</code></pre></div></div>
<p>For example : <code class="highlighter-rouge">ffmpeg [arguments] | ./ccaligner -useBuffer</code></p>
<p>The interface was modified to bring this change in place. You can try the various modes of reading in the VAD demo present in the <code class="highlighter-rouge">/demo</code> directory.</p>
<p>Though I try to document the code as I write it, there was still a way to improve it even more. I spent some time documenting the code, as well as the repository. I hope it is now even more easier to read and comprehend my code.</p>
<h3 id="how-did-the-evaluations-go">How did the evaluations go?</h3>
<p>The highlight of this week were the very first evaluations of my Google Summer of Code project. The evaluations window opened on 26th and were to remain open till 30th. Carlos (CCExtractor Org Admin, my mentor) alloted 28th as the day for my evaluations. My another mentor Alex is on his GCI trip to Google Office, SF.</p>
<p>This was the first time I was going to experience something like this, and no matter how much I researched or read about it, there’s no telling how the evaluations are gonna proceed and what is it going to involve. It basically just bottles down to mentor slash project slash work.</p>
<p>So, my evaluation began on 29th past Midnight (IST), i.e. 28th Morning at my mentor’s place. I must confess, I was super-nervous (which totally got reflected in my evaluations). So, Carlos sent me a DM on slack asking if I am ready for the evaluations, and I replied that I need few mins to push the final documentation that I wrote prior that day.</p>
<p>Within a span of 2 or 3 minutes I replied him back and the evaluation began. Carlos began by asking where can he obtain the binaries. I told him that he can compile his own (as I made sure it’s buildable across all platforms) or I can send him. He chose the first option and I told him how can he compile his own binaries, which basically boiled down to clonning the repo, and using make to build the <code class="highlighter-rouge">ccaligner</code> executable.</p>
<p>Now, I had a bit different picture about evaluations then what happened next. Sure, I expected him to test the code but I expected that more time would be spent on checking how the code actually is. Meaning, I was naive enough to expect that he’ll go through code file by file and see how things work, and is it good or not. I spent an awful amount of time making the code as flexible and adaptable as possible.</p>
<p>After building the tool, he chose a random sample from the HD he sent to us (with all the video samples), and run it against it. The tool worked as expected. But since it did not involve any audio processing (note : only approx aligner is implemented till this time in the interface), he asked me about it. I told him that the audio analysis part is being worked upon and he can try the components stored separetely (VAD) in the demo dir. He built the VAD demo and fed it the wave file obtained from the video file using FFmpeg and it gave the output in stdout with time frames and binary values - 0 for voice absent and 1 for voice present.</p>
<p>Then he proceeded to ask me if I have covered all the milestones I listed in my proposal and the checklist (<a href="http://saurabhshri.github.io/gsoc/">http://saurabhshri.github.io/gsoc/</a>). I gave him a brief overview against all the listed tasks in checklist and assured him that I have met them all. I asked him to be brutally honest because I wanted, rather I needed to hear what my mentor thinks about it.</p>
<p>This is what my mentor responded :</p>
<blockquote>
<p>I expected to see a bit more to be honest, but don’t be worried about it. If all the “cool previews” happen at stage 2 that’s fine. Your code quality is good, blog is good, communication is good… so no problems. OK, so eval done. you passed, so just continue working 🙂</p>
</blockquote>
<p>So, as you can see, though I passed my evaluations, I need to work even harder. I am happy that my mentor expects a lot from me and I hope in next evaluations I work upto his expectations.</p>
<p><img src="/images/posts/first_evaluations_result.png" alt="Official First Evaluations Result" /></p>
<p>In the official feedback, my mentor wrote :</p>
<blockquote>
<p>Code quality is good, however it would be useful to build in a way that allow to have good demos as work progresses. Building “completely horizontally” doesn’t allow to preview functionality. We’re betting on things working well at the end. Love the blog.</p>
</blockquote>
<p>I’ll continue to improve myself and learn more, as I have learned from this evaluation.</p>
<p>Thank you Carlos and Alex for being my mentor, and for passing me in the first evaluations! 😊 Let’s have another fun month of some open-source goodiness.</p>
<h3 id="how-about-the-stipend">How about the stipend?</h3>
<p>Ah well, looks like I was not very lucky at the stipend front 😛. Google was super-quick to release the stipend on 30th itslef. But for some reason, the payment for me got cancelled with the status <em>“Transfer rejected by processor.”</em> . Since it was weekend already, their support ass unavailable. Also, since it’s US holiday on 3rd and 4th, I can only hope that their Indian support is available on Monday. I emailed them about the issue and also sent and email to the GSoC team.</p>
<p>Looks like they were available on Monday, as the previous transaction was totally cancelled and a new transaction was made. I received an email that the money is sent to the bank, and shall be deposited within 4 to 5 days. Let’s hope for the best! 🙂</p>
<p><img src="/images/posts/gsoc_stipend_payoneer.jpg" alt="gsoc_stipend_payoneer.jpg" /></p>
<h3 id="whats-next">What’s next?</h3>
<p>Now I will begin working on implementing ASR in the tool. I am using CMU’s Pocketsphinx as it is light, portable and has great and active community. Plus, it’s in C, so should be easier to integrate it with my tool. Another possibility was to use Kaldi, but Kaldi in general is pretty resource demanding. Maybe I’ll add Kaldi support post GSoC. I have already started working on using Pocketsphinx’s API. I was successfully able to compile it using CMake with my tool and supply samples obtained from <code class="highlighter-rouge">read_wave_file.cpp</code> .</p>
<p>This time I might not make a very extensive task list as I made last time because this time I need to figure things on the go. I’ll keep posting the updates in this very blog.</p>
<p>I hope other students have passed their evaluations as well and are enjoying working. Remember guys, it can be frustrating at times and there will be a lot of factors trying to make your morale down. Fight it and you’ll emerge a winner. I hope I am able to do so as well. See you in the next one! 🎭</p>
<p><a href="https://saurabhshri.github.io/2017/07/gsoc/news/google-summer-of-code-week-4-the-evaluations">Google Summer of Code, Week 4 : The Evaluations! ⚰💀</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on July 03, 2017.</p>https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-3-printusage2017-06-23T00:00:00+05:302017-06-23T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Today marks the end of three of thirteen weeks of Google Summer of Code’s coding period. This week I implemented a simple user interface, added various output formats, fixed some bugs, fixed Unix build and processed samples.</p>
<blockquote>
<p>All the latest commits could be found in the <code class="highlighter-rouge">development</code> branch. I will merge them to <code class="highlighter-rouge">master</code> once the first phase completes.</p>
</blockquote>
<p>The code written till now is arranged systematically and is in the form of library so that anyone can use it in her code and make use of the available functionalities.</p>
<p>Say, for example, you want to make use of Approx Aligner (to perform approximate word by word audio subtitle synchronization) in your own cool project. All you need to do is, clone the CCAligner repository and put it in your project directory. Then simply include the “generate_approx_timestamp.h” file in your project by providing appropriate path. Also, include all the other files in your CMake file, and you are good to go! You can then simply do <code class="highlighter-rouge">ApproxAligner * aligner = new ApproxAligner(input_filename);</code> and use appropriate functions for your use. 😎</p>
<p>I am trying to include all the examples in the <code class="highlighter-rouge">/demo</code> directory in the repository. You can find it <a href="https://github.com/saurabhshri/CCAligner" title="CCAligner Demo">here</a>. They have their own CMakeLists.txt in the respective directories so that you can build them individually if you want to. I will add a demo install script later in which you should be able to pass the demo you want to see as a parameter and it’ll build it for you.</p>
<p>While it’s great for developers to use this, there was no way to use it just out of the box. So, I have built a small user interface so that it should be easy to use and try and maybe use it as a base to develop your application. This enables using the tool directly after cloning it and then building it. The interface is pretty minimal and is command line only. You can run the tool without any arguments to see the available options. I will be updating the readme file with the instructions once it is complete.</p>
<p>Here’s a snippet of the same :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ApproxAligner : ccaligner -a input.srt
ccaligner -a input.srt -of <output_format>
(srt/xml/json/stdout)
e.g. ccaligner -a input.srt -of xml
</code></pre></div></div>
<p>As evident from the above usage examples, there are now options available to have output in more than just SRT format. The output can currently be obtained in the form of SRT, XML, JSON format and also on stdout. The structure of all of them is shown below. The respective functions can be edited and extended to user desired scheme.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
/* printToXML() */
├───Subtitle
├───Start Time
├───End Time
├───Start Time in ms
├───Start Time in ms
├───Word
├───Style
| ├───Style Tag 1
| ├───Style Tag 2
| └───Style Tag
Example :
<subtitle>
<time start=3560 end=3805></time>
<srtTime start=00:00:03,560 end=00:00:03,805></srtTime>
<text>It's</text>
</subtitle>
<subtitle>
<time start=3805 end=4099></time>
<srtTime start=00:00:03,805 end=00:00:04,099></srtTime>
<text>great</text>
</subtitle>
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
/* printToConsole() */
START
word : {word here}
start : {start time in ms}
end : {end time in ms}
END
Example :
START
word : It's
start : 3560 ms
end : 3805 ms
END
START
word : great
start : 3805 ms
end : 4099 ms
END
</code></pre></div></div>
<p><img src="/images/posts/output_formats.png" alt="output_formats.png" />
🔥</p>
<p>Also the Unix build (Linux and MacOS) was failing ever since I added the VAD. The <code class="highlighter-rouge">webRTC</code> library failed to link. Turns out, webRTC uses <code class="highlighter-rouge">WEBRTC_POSIX</code> variable to determine if the system is POSIX. But for some reason, this variable did not get set. It wasn’t apparent at first that this is the problem. After a lot of digging deep I found that this was the culprit. I had to add a manual check for posix systems so that correct multithreading code executes in respective environment.</p>
<p>Once this was fixed, it turned out while the pthread library successfully links in MacOS, it wasn’t linking it in Linux. Supplying <code class="highlighter-rouge">-lpthread</code> as a flag while linking solved the problem. The CMakelist.txt was modified to incorporate all these changes. The same was reflected in the VAD demo as well.</p>
<p>Also, there’s a special requirement of type of wave files that are needed by ASR and VAD library. The wave file must be 16 bit PCM, Mono, 16Khz or 8Khz sampled. So, I processed the samples to obtain the wave files.</p>
<h3 id="whats-next">What’s next?</h3>
<p>I originally planned to do the video chunking, i.e. splitting the videos such that they only contain the parts where audio is present. But for now I am planning to postpone it as it totally relies on the way that will be best suited as per ASR’s requirement. So, I will do it at that time.</p>
<p>I will now add the capability to read the wave file from stream i.e. in a way that data can be piped through other programs such as ffmpeg. So that doing something like <code class="highlighter-rouge">ffmpeg -i video <other params> | ccaligner </code> will be possible. I will also be working on adding sliding window approch to VAD to get better results. It should be interesting to see if it improves the accuracy.</p>
<p>I will be preparing a full report of the progress so far, complete the documentation, update the same on ccextractor.org et cetera.</p>
<p>Also, 28th is the date when Carlos has decided to read the report so that he can submit the evaluation. The evaluations start on 26th and are due before 30th. I am a bit nervous (bit >= 64) as this is the first evaluation ever I am facing in my entire life. I hope everything goes good. I am very happy with my mentors, they are super nice and super fun to work with. I am glad that they trusted me enough to be in the driver’s seat for this project.</p>
<p>All the very best to all the GSoC participants for their very first evaluations. I hope you all (inlcuding me) pass with flying colours! 🌈</p>
<p><a href="https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-3-printusage">Google Summer of Code, Week 3 : printUsage(); 🖨</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on June 23, 2017.</p>https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-2-valar-researchis2017-06-18T00:00:00+05:302017-06-18T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>The second week of Google Summer of Code just wrapped up. This week was spent on adding support for managing timestamps for each word, improving Approx Aligner, printing result as SRT, revising Game of Thrones and doing a lot of reading and researching. The first evaluations are due in two weeks.</p>
<blockquote>
<p>I had imagined that the Approx Aligner would be far from accurate, the results are definitely not bad. I have attached the demo later on in this post.</p>
</blockquote>
<p>The first evaluations are even more close as the third week of coding period begins. Since the beginning of the program I am properly (rather ahead atm) on the <a href="https://saurabhshri.github.io/gsoc/">timeline</a> I proposed. Hence, I did not write <em>much</em> code in this week, but instead I spent the time learning few things which will be helpful for the project later on. Another reason for that was the fact that I was sick (fever) for a while. 🤒</p>
<p>Here’s the summary of things I did in the second week :</p>
<ol>
<li>
<p>Added support for managing the timestamps of individual words.</p>
<p>The parser was modified to add the support of storing the start-time and end-time of each word ad accessing it. The time values are stored as a vector of <code class="highlighter-rouge">long int</code> and there are functions to get these values both as vector or individual values by index.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
//Relevent Data Members :
std::vector<std::string> _word; //list of words in dialogue
std::vector<long int> _wordStartTime; //start time of each word in dialogue
std::vector<long int> _wordEndTime; //end time of each word in dialogue
std::vector<long int> _wordDuration; //actual duration of each word without silence
//Relevent Member Functions :
std::vector<std::string> getIndividualWords(); //return string vector of individual words
std::string getWordByIndex(int index); //return word stored at 'index'
std::vector<long int> getWordStartTimes(); //return long int vector of start time of individual words
std::vector<long int> getWordEndTimes(); //return long int vector of end time of individual words
long int getWordStartTimeByIndex(int index); //return the start time of a word based on index
long int getWordEndTimeByIndex (int index); //return the end time of a word based on index
</code></pre></div> </div>
<p>This makes things very easy. For example printing the words and their timestamps in SRT format is now as easy as doing :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> //converting STARTING timestamp (in milliseconds) to hh:ss:mm,ms
ms_to_srt_time(_sub->getWordStartTimeByIndex(i),&hh1,&mm1,&ss1,&ms1);
//converting ENDING timestamp (in milliseconds) to hh:ss:mm,ms
ms_to_srt_time(_sub->getWordEndTimeByIndex(i),&hh2,&mm2,&ss2,&ms2);
//arranging the timestamps in SRT style
sprintf(timeline, "%02d:%02d:%02d,%03d --> %02d:%02d:%02d,%03d\n",
hh1, mm1, ss1, ms1, hh2, mm2, ss2, ms2);
out<<timeline;
out<<_sub->getWordByIndex(i)<<"\n\n";
</code></pre></div> </div>
</li>
<li>
<p>Improving the Approx Aligner and making output as SRT</p>
<p>Using the word weight to find it’s approximate timeframe is bound to have errors. I spent the time in figuring out how can I still improve it and if its even possible. After a lot of experimentation it hit me - apart from the words, the one other thing that makes a difference in actual speech is the “silence” between those words which I was not considering at all. So, using word weight and finding the duration of each word, I split the remaining time into the silence between each word being spoken. One more addition could be a more emphasis on silence in presence of punctuation, but this is difficult to calculate as it hugely varies from speaker to speaker and situation to situation (well, so does other things).</p>
<p>Anyway, after making changes, I also added the option to print the result as an SRT file so that I can actually see the results and verify them. I will make a report on it’s accuracy once I run it across all the samples. The disk containing samples will reach to me soon. The results right now, surprisingly, <strong>are not bad at all!</strong></p>
<p>I tested it on couple of samples, and it was pretty great given that the method uses 0% audio processing and how fast it is and that it’s entirely based on probability and approximation! Here’s a quick screengrab demostrating the result :</p>
<div style="position:relative;height:0;padding-bottom:56.25%"><iframe src="https://www.youtube.com/embed/km1iHe_mGuo?ecver=2" style="position:absolute;width:100%;height:100%;left:0" width="640" height="360" frameborder="0" allowfullscreen=""></iframe></div>
<p>You can see the results yourself too. Here are the video sample and it’s <em>word by word synced</em> subtitles generated using Approx Aligner. :)</p>
<ul>
<li>Video File : <a href="http://gsocdev3.ccextractor.org/~saurabhshri/repository/ted/ApproxAligner/ElonMusk2017.mp4">ElonMusk2017.mp4</a></li>
<li>Subtile File : <a href="http://gsocdev3.ccextractor.org/~saurabhshri/repository/ted/ApproxAligner/ElonMusk2017.srt">ElonMusk2017.srt</a></li>
</ul>
<p>This is actually pretty good, because now we will be able to have a window (say +- {ms} of this timestamp) where the word will actually be present and hence we can focus our ASR to detect the word in this range rather than trying to guess it over a period of time.</p>
</li>
<li>
<p>Loads of reading and researching 📑</p>
<p>I did not want to waste any time just because I was sick. So, I got few research papers in the field of Forced Aligners and ASR printed and spent time reading them. Also, I read the code of the programs using these ASRs to see in what ways people are using them to accomplish their tasks.</p>
<p>I now have an even better picture of how and what I am going to do in phase two. Also, I already have the capability to perform VAD and read audio samples implemented ahead of time which is a plus.</p>
</li>
</ol>
<h3 id="whats-next">What’s next?</h3>
<p>I will be implementing the various other output options, such as JSON and XML. Also, I’ll refactor somme code and write documentation on using the Approx Aligner. The tool is in the form of libraray currently and it makes sense to also provide an interface so that it can be used directly as well (i.e. without writing code), after all - it’s a <em>tool</em>. I should have the disk containing samples soon - so I’ll have to process it to get audio in the required format from them and also work on some batch scripts to automatically run test over all of them.</p>
<p>This time of summer is quite daunting here in India. It’s hot, but worse, it’s humid. My eyes need a checkup as well - I am often having headaches and blurry vision (not a good sign I know). I hate spectacles and definitely don’t want them. The last time I had check-up, the doctor pointed out that my right eye needs a glass of 0.5 power. But given my hatred towards them I did not get my glasses made. I sincerely hope it’s not worsened.</p>
<p>Anyway, I hope everyone’s having fun and the other participants are having great time building their projects. ⛱</p>
<p><a href="https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-2-valar-researchis">Google Summer of Code, Week 2 : Valar Researchis! ⚔</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on June 18, 2017.</p>https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-1-the-beginning2017-06-09T19:30:00+05:302017-06-11T00:00:00-00:00Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>It has been two days since Google Summer of Code’s first of thirteen weeks ended. I have started working on my project CCAligner - Word by Word Audio Subtitle Synchronization with CCExtractor Development.</p>
<blockquote>
<p>My progress could be tracked through weekly checklist for milestones and tasks, which can be accessed here : <a href="https://saurabhshri.github.io/gsoc/">https://saurabhshri.github.io/gsoc/</a> .</p>
</blockquote>
<p>With the <a href="https://saurabhshri.github.io/2017/05/gsoc-how/news/gsoc-2017-end-of-community-bonding-period">end of Community Bonding Period</a>, the <a href="https://saurabhshri.github.io/2017/05/gsoc/gsoc-2017-coding-period-begins">coding period officially began</a>. This is how I spent the first week of coding period :</p>
<ol>
<li>
<p>Setting-up the new server. 💻</p>
<p>As I mentioned in the last post, I was already done with setting-up my development environment. But I had made a request for a dedicated server of my own with root access and Carlos (CCExtractor Org admin, my co-mentor) got me one! :) It’s a nice little server running Ubuntu Server. This was the first time I used such a vanilla version of linux. It’s super light out of the box and has absolutely nothing (just the linux) installed. It was so much fun setting up the server according to my preferences.</p>
<p>I also installed x2go in it so that I can also “visually” see the changes I am making. It’s super-helpful while debugging.</p>
</li>
<li>
<p>Started building basic skeleton of tool.</p>
<blockquote>
<p>CCAligner can be found on Github at : <a href="https://github.com/saurabhshri/CCAligner/" title="CCAligner - word by word audio subtitle synchronisation">https://github.com/saurabhshri/CCAligner/</a></p>
</blockquote>
<p>I made a rough sketch of how the general hierarchy will be in the project. I will try to make my tool as modular as possible, and also in the way that it can be easily used as a library in other projects. It’s often difficult to “librarise” the code after creating, so even if it consumes some extra time, it’s better in my belief to begin doing the same.</p>
<p>In general, there are two main directories inside the <code class="highlighter-rouge">source</code> directory -</p>
<ul>
<li><code class="highlighter-rouge">lib_ext</code> : This shall contain all the <a href="https://github.com/saurabhshri/CCAligner/tree/master/src/lib_ext/">external libraries</a> that I’ll use.</li>
<li><code class="highlighter-rouge">lib_ccaligner</code> : This shall contain the <a href="https://github.com/saurabhshri/CCAligner/tree/master/src/lib_ccaligner">CCAligner library</a>.</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ├───demo
│ ├───ApproxAligner
| └─── ...
├───src
│ ├───lib_ccaligner
| | ├─── ...
│ | └─── ...
│ └───lib_ext
│ ├─── ...
│ └─── ...
└───tests
├─── ...
└─── ...
</code></pre></div> </div>
</li>
<li>
<p>Begin writing the implementation of approximation based word tagging.</p>
<p>Using the <a href="https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp">subtitle parser I created</a>, I began writing the implementation of approximating the timestamp of a word based on it’s weight (calculated as a function of ratio of word length to sentence length). This is a super fast method to calculate the timestamp of each word and doesn’t require any type of audio processing. But obviously, this has very poor accuracy. Nonetheless this shall come in handy where audio - analysis is <em>not</em> feasiable and we require the sync super-fast without caring much about accuracy. Not to forget, this will provide us with a better window to analyse the word when audio analysis will come into the picture.</p>
<p>Since, the code is in the form of library, it’s very easy to use. The demo can be found in the <code class="highlighter-rouge">demo</code> directory which is linked <a href="https://github.com/saurabhshri/CCAligner/tree/master/demo" title="Approx Alignment Demo.">here</a>.</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cp">#include "generate_approx_timestamp.h"
</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="o"><<</span><span class="s">"Enter path to the subtitle file : "</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">filename</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">cin</span><span class="o">>></span><span class="n">filename</span><span class="p">;</span>
<span class="n">ApproxAligner</span> <span class="o">*</span> <span class="n">aligner</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ApproxAligner</span><span class="p">(</span><span class="n">filename</span><span class="p">);</span> <span class="c1">// that's it :) More customization to come!
</span> <span class="n">aligner</span><span class="o">-></span><span class="n">align</span><span class="p">();</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div> </div>
<p>It has room for ton of improvement, which I definitely will do with the course of time. Everything is currently pretty raw, as you’ll expect a software to be in it’s primary stages. And yes, I’ll begin naming my header files in a better way! :P</p>
</li>
<li>
<p>Set-up a small testing environment. ✔️</p>
<p>As mentioned in the last post, I have collected quite a lot of samples, a mojority of which are Ted talks (as they have clear speech as well as good subtitles). Previously I worked on <a href="https://github.com/canihavesomecoffee/sample-platform/commit/4b6cc37ab9bbcce81e5c78f6dbe5dffd297f5ca8">improving CCExtractor Sample-Platform’s difference showing library</a>. I extended this so that I can compare the output of the tool with the actual results, and also to compare results across various techniques.</p>
<p>I have also set-up Travis-CI to check builds across linux and OSX with every commit to the repository. Soon, I’ll add the test scripts to it as well. This will be a step towards test driven development.</p>
</li>
<li>
<p>Start implementation for audio based processing.</p>
<p>The very first step in this region was being able to read audio files and extract data. For this my code should be able to take wave file as input, check if the file is valid or not, and then decode it to extract information such as SampleRate, BitRate, etc. and the samples (which contain the audio data) itself. This was an intresting job. While I could find some code online that did the <em>reading</em>, but they were not utilising the object oriented approach and also, did not read the data in the manner desired for my use.</p>
<p>So, I searched for the wave file specifications online. The official Microsoft’s document was not the example of best specification-documentation I have seen, but was OK. Fortunately, I found another one which was very well written, and was precise. If you are interested, it is located <a href="http://soundfile.sapp.org/doc/WaveFormat/" title="Wave File Specifications">here</a>.</p>
<p>I did encounter few bugs while decoding wave files. Though the bugs were pretty small and easily fixable, they took some time to get spotted.</p>
<p>1.When I read the wave files in buffer, and used it to create a new wave file, for no apparent reason, the output was a very scattered and noisy version. Now, it felt pretty ridiculous to me that how can the output be different if all I am doing is reading it as it is and sending it as output.</p>
<p><img src="/images/posts/reading_wav_file_cpp.jpg" alt="Difference in input and output wave files." />
<em>Difference in input and output wave files.</em></p>
<p>After <a href="https://www.quora.com/What-is-the-most-interesting-bug-you-have-ever-solved-in-a-computer-program/answer/Sean-Lyndersay" title="I even started blaming compiler!">spending some time on it</a>, I ended up opening both the input and output wave files in a hex viewer and search for the differences. After few bytes, I could see that the hex code shifted by one, i.e.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 632e ca30 0a2d 582f 4c2b 8c2d 032a 2b2c
632e ca30 0d0a 2d58 2f4c 2b8c 2d03 2a2b
</code></pre></div> </div>
<p>Now, it became clear. Everywhere in the file, <code class="highlighter-rouge">0a</code> was replaced by <code class="highlighter-rouge">0a0d</code>. Now this immediately raised a question in my mind. Could it be..? I <a href="https://www.google.co.in/search?q=0a+replaced+by+0a0d&oq=0a+replaced+by+0a0d&aqs=chrome..69i57.5652j0j7&sourceid=chrome&ie=UTF-8" title="0a replaced by 0a0d">Googled it</a>, and yes! It was the endline characters. All the <code class="highlighter-rouge">LF</code> were converted to <code class="highlighter-rouge">CRLF</code> leading to the distortion in the output file. Turn out, while I was reading the file in binary format, I was writing it in the text format, which led to this fiasco. Once the bug found, setting the <code class="highlighter-rouge">ios::binary</code> flag was all it took.</p>
<p>2.Another intresting thing that happened was, everything was being decoded perfectly, <a href="https://github.com/saurabhshri/CCAligner/commit/aa5c9aa33be37777e812f1d7d87a30ab5caba49e" title="Fixing negative values.">except the SampleRate</a>. This was strange because it was in the middle of various value - which were decoded perfectly. I had to read the detailed specification to find out that the data is in the form of unsigned values. I had not considered it and I was storing it in signed char. So, where the values should be, say for example, 255 it stored -128.</p>
<p>3.The specs which I used for reference were incomplete. It missed the fact that the wave file is not necessarily 44 Bytes and that it may contain some meta-data. So, to find the <em>data subchunk</em>, I <a href="https://github.com/saurabhshri/CCAligner/commit/e421fb2baac72f615647f49bbac2d45b09669b8f" title="Compensating for presence of metadata in the wave file header">manually searched for them</a> in the stream.</p>
<p>At the end, the <a href="https://github.com/saurabhshri/CCAligner/commit/66f249027fa653d25e4045c95c5178d07c3616d2">wave files are now being correctly read</a>, and the the samples are collected as a vector of 16 bit integers (unsigned short int) : <code class="highlighter-rouge">vector<int16_t> _samples</code>.</p>
</li>
<li>
<p>Read about various available VAD techniques.</p>
<p>I researched about various available VAD techniques, compared their pros and cons and at the end decided to go with Google’s webRTC. They have one of the best VADs out there and the code is already in C. It’s documentation to use the native C code is almost non-existant. While I was able to locate the function that performs VAD by peeking into the files myself, it was not very clear about the arguments that needed to be supplied.</p>
<p>I did search the <em>discuss-webRTC</em> Google group, but there was nothing to be found in the forum. I even asked them on their IRC channel, but I was instructed to ask the question on StackOverflow, which I did. I was not very sure that someone would answer and when I saw the stats for webRTC related question, that increased my suspicion.</p>
<p><img src="/images/posts/unanswered.PNG" alt="Unanswered questions in webRTC tag." /><br />
<em>68.4% Questions Unanswered in last 30 days.</em></p>
<p>Fortuantely I found out that there is a python wrapper of webRTC’s VAD in the Github. I immediately sent an email to <a href="https://github.com/wiseman">John Wiseman</a>, who created the awesome <a href="https://github.com/wiseman/py-webrtcvad">py-webrtcvad</a> and asked him if he could help. Now, I wasn’t sure that I would find a reply. So, I looked into the SO stats to see the “most active” members on the webRTC tag and also emailed two people from there.</p>
<p>I was surprised to find that all the three people replied and all were willing to help. This is what I love about the open source community, everyone is eager to help. ❤️ Sometimes this comes rather surprising to me. Thank you John, Ajay and Gilad. :)</p>
<p>So ultimately John answered <a href="https://stackoverflow.com/q/44457162/6487831">my question on SO</a>, and I have a small implementation of VAD ready. In the coming weeks I’ll improve it and bring it to use.</p>
</li>
<li>
<p>Fixed the CMake build script of CCExtractor for windows.</p>
<p>CCExtractor had broken CMakeLists.txt since a long time. Due to this, the only way to compile it on Windows was using Visual Studio. I prefer to work in CLion so I spent some time and <a href="https://github.com/CCExtractor/ccextractor/commit/51d936bc9076e131b4e8b089c664251c3d2f2861">fixed the CMake list</a>. It should also be helpful in bringing the Windows support to Sample-Platform which can use this to build it.</p>
</li>
</ol>
<h2 id="whats-next">What’s next?</h2>
<p>This was the first of total thirteen weeks. There’s a lot of work left. The work already done is also pretty raw and there’s a room for tons of improvement. My mentors have suggested a few changes as well. E.g. The tool needs to be able to read directly from the stream (like the data being piped from FFMPEG). I will stick to the timeline and continue the work. I am in constant contact with my mentors and shall work in accordance with their feedbacks.</p>
<p>The second week has already commenced and the evaluations are in the June end. I hope other GSoC participants are enjoying their work as well! 📈</p>
<p><a href="https://saurabhshri.github.io/2017/06/gsoc/google-summer-of-code-week-1-the-beginning">Google Summer of Code, Week 1 : The Beginning! 🕺</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on June 09, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc/gsoc-2017-coding-period-begins2017-05-29T19:30:00+05:302017-05-29T19:30:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>This marks the ending of a month long community bonding period. Fine-tuning deliverable, setting up timeline, early coding and much more, here’s my GSoC 2017 community bonding experience with CCExtractor Development. 😊</p>
<blockquote>
<p>As it turns out, the little backstory written below is not <em>little</em> afterall. 😛 <a href="#communinty-bonding-period" title="Community Bonding Period">Click here</a> to skip it and jump to later portion.</p>
</blockquote>
<h3 id="a-little-backstory">A little backstory</h3>
<p>I began preparing for GSoC in November and at it took me no time to decide that CCExtractor is the one I would love to work with. It was one of the first orgs I researched about and I immediately liked them. They were amidst Google Code-In and it was actually pretty awesome to watch the participants and mentors working together.</p>
<p>I distinctly remember the starting few days. Though I had joined CCExtractor’s discussion group on Slack, I did not say anything in the beginning. I only stayed their and read their conversations. It was not because I didn’t have anything to say or that I did not want to introduce myself, but because I was hesitant. I had never done this before. I was dead scared, and nervous. But then I saw how lovely the community is. The Code-In participants were in the age group of 13-18 and seeing them work together gave me a boost in confidence. The mentors were super - supportive. Even when someone was being PITA they handled it pretty flawlessly. So, I thought, if these kids can do it, why can’t I (though in reality, they were completing Code-In taks and I was merely trying to introduce myself 😛). So, I introduced myself with my first PR which was a mere correction of a typo in a readme file. I was surprised to see how welcoming everyone was. Not only the mentors, even the participants gave me warm welcome. My hesitation went from 100% to 0% in no time.</p>
<p>That was it. From that moment I actively started participating in the discussions, code reviews, random chats and what not. I started contributing and also helped the code-in students in verifying their code and helping them test things. Along with the main CCExtractor tool, I also had a lot of fun working on Sample-Platform! Also, contributing to sample-platform taught me a great deal of things - not just talking about code - but other important things like open-source ethics, working with people, making good PRs, asking good and quality questions, doing research, proper version control et cetra. Thank you Carlos and Willem, I can not thank you both enough! During this period I also became good friends with Alex and Evgeny who were code-in participants (and winners) and now GSoC mentors. It’s super-fun to work with all of these guys and its awesome to be a part of the community!</p>
<h3 id="communinty-bonding-period">Communinty Bonding Period</h3>
<p>The amazing news of <a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017">my proposal getting selected</a> for this GSoC was announced on 4th of May which also marked the begining of community bonding period. Community bonding is basically a month allocated to learn about one’s organization’s processes - release and otherwise - developer interactions, codes of conduct, et cetera. What once choose to do during this period varies.</p>
<p>Since I was already active in the community since quite some time, I was already familiar with most of the people and with the general practices. I have two mentors - <a href="https://github.com/cfsmp3">Carlos Fernandez Sanz</a> (who originally built CCExtractor) and <a href="https://github.com/AlexBratosin2001">Alex Bratosin</a> (CCExtractor GCI 2016 Winner).</p>
<p>So here are the things I did during the community bonding period.</p>
<ol>
<li>
<p>Set-up a blog.</p>
<p>This was actually quite important. To note down my GSoC progress and to keep mentors and everyone in the loop, I created this blog. It’s actually a simple Jekyll blog hosted on Github Pages.</p>
<p>Blog link : <a href="https://saurabhshri.github.io/">https://saurabhshri.github.io/</a>.</p>
<p>Additionally I decided that it would be easier for my mentors and me to keep a check on work if we have a detailed checklist of the tasks which is to be done. My mentors agreeed, and hence I am maintaining the milestones and deliverable in the form of a checklist as Github Gist, so that it’s easily embeddable.</p>
<p>Milestones/Weekly deliverable checklist : <a href="https://saurabhshri.github.io/gsoc/">https://saurabhshri.github.io/gsoc/</a> .</p>
<p>Till now I have written about three posts (fourth including this) about my GSoC work on the blog the first being <a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017">announcement of me getting accepted</a>. All my GSoC work progress and related work is tagged under the “Gsoc” category.</p>
<p>Category wise posts : <a href="https://saurabhshri.github.io/categories/">https://saurabhshri.github.io/categories/</a> .</p>
<p><img src="/images/posts/blog_screenshot.PNG" alt="" />
<em>Screenshot of one of the blog posts</em></p>
</li>
<li>
<p>Brought myself a chair.</p>
<p>Students participating in GSoC are expected to work at least 40 hours a week, and never having done anything for such a long time before, it was an easy realization that a proper chair is a must. I brought a nice ergonomic chair with lambar support and adjustable height and it really helps in the backpain caused due to bending in front of laptop. I think it was worth an investment.</p>
</li>
<li>
<p>Completed GSoC formalities including setting up Payoneer account.</p>
<p>This took more efforts than I originally expected. Setting up mode of payment (i.e. deciding to recieve money in USD or INR) took some significant amount of time. I visited several banks to enquire about their fees. Plus when I decided and set-up the method, it was chosen wrongly dure to some error on Payoneer’s end. I had to again send some documents and information to them on email, after around two weeks of email interchanges the account was finally set-up.</p>
<p><img src="/images/posts/payoneer.PNG" alt="" /><br />
<em>Google added as funding source in Payoneeer account</em></p>
<p>I will post a detailed post about setting up payment process to help future students sometime soon.</p>
</li>
<li>
<p>Fine tuning deliverable.</p>
<p>I spent a lot of time reading about the techniques I might incorporate while doing the project. Though I had made a pretty detailed plan while making the proposal, I fine tuned the deliverable for first phase. They can be found in the checklist I mentioned above.</p>
</li>
<li>
<p>Creating sample repository.</p>
<p>The kind of samples that are required for my tool are a bit restrcitive. Two of the primary conditions are :</p>
<p>a. samples must have subtitles<br />
b. samples must have clear speech audio</p>
<p>To find such samples I enquired various people like creator of Kaldi based forced-aligner Gentle and CMUSphinx community. Nikolay from CMUSphinx gave me a very good idea to use Ted Talks as sample. They fit both my primary requirements and are easily available to download as well.</p>
<p><img src="/images/posts/sample.PNG" alt="" /><br />
<em>Some Ted samples</em></p>
<p>Also, Carlos is sending a 2 TB disk loaded with transport streams recorded from various TV stations. Let’s see when I get to recieve that.</p>
</li>
<li>
<p>Started coding early.</p>
<p>I decided to start coding early and hence started to work on a subtitle parser which is one of the primary requirement of my tool. I have already completed it (at least the part required in my project).</p>
<p><img src="/images/posts/srtparser.PNG" alt="" /><br />
<em>Subtitle Parser</em></p>
<p>Link to parser : <a href="https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp">https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp</a> .</p>
<p>Link to relevant blog posts :</p>
<p>a. <a href="https://saurabhshri.github.io/2017/05/gsoc/creating-a-full-blown-srt-subtitle-parser">Creating a full blown (SRT) Subtitle Parser </a></p>
<p>b. <a href="https://saurabhshri.github.io/2017/05/gsoc/simple-yet-powerful-single-header-srt-subtitle-parsing-library-in-cpp">Simple yet powerful single header srt subtitle parsing library in cpp</a></p>
<p>I have also begun working on setting up testing scripts which will help me during coding period.</p>
</li>
<li>
<p>Researching</p>
<p>This is something which I will need to do throughout. A lot of things which I am planning to do in my project like VAD, ASR, Phoneme Recognition, LM Training et cetera are something which are new to me. It’ll be interesting to learn about them and implement them. I am quite excited and a little nervous about them. :)</p>
</li>
<li>
<p>Setting up development environement.</p>
<p>I inititally hoped to recieve a seperate server to work upon, but I understand that it is not entirely possible to give me my own separate server. I will be working on the gsocdev3 server on which I used to work before. It’s an excellent machine. The internet is quite good and it has huge sample collection. I also have my laptop with several VMs all ready to test the code I write. I have also enrolled myself in AWS free tier in case I need it.</p>
</li>
</ol>
<h3 id="whats-next">What’s next?</h3>
<p>The coding period officially commences from 30th May. I will stick to the timeline and work to tick those square boxes in the checklist. I will try to complete the tasks ahead of time in order to save time in case I get stuck at some point (which is often the case developing something new). Also, first phase evaluations begin on 26 June.</p>
<p>I wish all the participants a fun and productive summer. 🙋</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc/gsoc-2017-coding-period-begins">GSoC 2017, Coding Period Begins! ⚡</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 29, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc-how/news/gsoc-2017-end-of-community-bonding-period2017-05-29T15:10:00+05:302017-05-29T15:10:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Community bonding period, where students spend time learning and getting ready for their project, ends on 30th of May.</p>
<p>The Google Summer of Code 2017 community bonding period began with the announcement of <a href="https://summerofcode.withgoogle.com/projects/">accepted projects</a> on 4th of May and will end tomorrow, on 30th of May. This will mark the <strong>official</strong> commencement of Coding Period, which will last for 3 months. Students shall begin writing code for their projects under their mentors based on the timeline and milestones they set during this community bonding period. First evaluations are scheduled to take place from 26th June till 30th June.</p>
<p><img src="/images/posts/community_bonding.PNG" alt="" /></p>
<p>I will post about my community bonding experience and the work progress in my next blog post soon. :) Meanwhile you can find my profile on Github here : <a href="https://github.com/saurabhshri">https://github.com/saurabhshri</a> and my GSoC timeline and deliverable checklist here : <a href="https://saurabhshri.github.io/gsoc/">https://saurabhshri.github.io/gsoc/</a> .</p>
<p>I wish all the best too the GSoC 2017 participants (including me). Let’s make something awesome this summer!</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc-how/news/gsoc-2017-end-of-community-bonding-period">GSoC 2017 : End of Community Bonding Period</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 29, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc/simple-yet-powerful-single-header-srt-subtitle-parsing-library-in-cpp2017-05-29T11:11:00+05:302017-05-29T11:11:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Srtparser.h : Simple, yet powerful single header C++ SRT Subtitle Parser Library. 💖</p>
<p>This is a follow-up blog post to my <a href="https://saurabhshri.github.io/2017/05/gsoc/creating-a-full-blown-srt-subtitle-parser">previous post</a>, where I began implementing a fully functional but simple to use subtitle parser in C++ for my <a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017">GSoC project</a>.</p>
<p>I am happy to announce that the subtitle parser is ready. <strong>You may access it here</strong> : https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp</p>
<p>I have tried my best to document it in the Github repo, and it should be fairely easy to use. But in case you have any doubt or you need any help, feel free to contact me or raise an issue in that Github repo, I will be happy to help. 😁</p>
<p>The parser is super easy to use and has tons of features ✨ :</p>
<ul>
<li>It is a single header C++ (CPP) file, and can be easily used in any project.</li>
<li>It is focused on portability, efficiency and simplicity and has no external dependency.</li>
<li>Wide variety of functions at programmers’ disposal to parse srt file as per need.</li>
<li>Some amazing and useful capabilities such as :
<ul>
<li>extracting and stripping HTML and other styling tags from subtitle text.</li>
<li>extracting and stripping speaker names.</li>
<li>extracting and stripping non dialogue texts.</li>
<li>extracting words as list.</li>
<li>get time in both string and in milliseconds.</li>
</ul>
</li>
<li>It is super easy to extend and customize.</li>
</ul>
<p>I could not find any other subtitle parser / parsing library which was apt for my usage and I ended up creating one myself. Hope this is helpful to other developers as well. This is definitely one of the best ones out there. Feel free to use it.</p>
<h1 id="how-to-use-srtparserh-subtitle-parser">How to use srtparser.h subtitle parser</h1>
<ol>
<li>
<p>Include the header file in your program.</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cp">#include "srtparser.h"
</span></code></pre></div> </div>
</li>
<li>
<p>Create SubtitleParserFactory object. Use this factory object to create SubtitleParser object.</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">SubtitleParserFactory</span> <span class="o">*</span><span class="n">subParserFactory</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SubtitleParserFactory</span><span class="p">(</span><span class="s">"inputFile.srt"</span><span class="p">);</span>
<span class="n">SubtitleParser</span> <span class="o">*</span><span class="n">parser</span> <span class="o">=</span> <span class="n">subParserFactory</span><span class="o">-></span><span class="n">getParser</span><span class="p">();</span>
</code></pre></div> </div>
</li>
<li>
<p>Use the parser. 😉</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">SubtitleItem</span><span class="o">*></span> <span class="n">sub</span> <span class="o">=</span> <span class="n">parser</span><span class="o">-></span><span class="n">getSubtitles</span><span class="p">();</span>
<span class="kt">long</span> <span class="kt">int</span> <span class="n">startTime</span> <span class="o">=</span> <span class="n">sub</span><span class="err">→</span><span class="n">getStartTime</span><span class="p">();</span>
</code></pre></div> </div>
</li>
</ol>
<p><strong>Read more about the available functions in this easy to read and explanatory table</strong> : <a href="https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp#parser-functions">https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp#parser-functions</a> .</p>
<p>You may also checkout a demo program using this library in <code class="highlighter-rouge">example/</code> directory.</p>
<h3 id="elements-present-in-subtitle---item">Elements present in subtitle - item</h3>
<p>Following is the list of all fields present in subtitle item :</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kt">long</span> <span class="kt">int</span> <span class="n">_startTime</span><span class="p">;</span> <span class="c1">//in milliseconds
</span>
<span class="kt">long</span> <span class="kt">int</span> <span class="n">_endTime</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">_text</span><span class="p">;</span> <span class="c1">//actual line, as present in subtitle file
</span>
<span class="kt">long</span> <span class="kt">int</span> <span class="n">timeMSec</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">value</span><span class="p">);</span> <span class="c1">//converts time string into ms
</span>
<span class="kt">int</span> <span class="n">_subNo</span><span class="p">;</span> <span class="c1">//subtitle number
</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">_startTimeString</span><span class="p">;</span> <span class="c1">//time as in srt format
</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">_endTimeString</span><span class="p">;</span>
<span class="kt">bool</span> <span class="n">_ignore</span><span class="p">;</span> <span class="c1">//should subtitle be ignore; used when the subtitle is empty after processing
</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">_justDialogue</span><span class="p">;</span> <span class="c1">//contains processed subtitle - stripped style, non dialogue text removal etc.
</span>
<span class="kt">int</span> <span class="n">_speakerCount</span><span class="p">;</span> <span class="c1">//count of number of speakers
</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">_speaker</span><span class="p">;</span> <span class="c1">//list of speakers in a single subtitle
</span>
<span class="kt">int</span> <span class="n">_nonDialogueCount</span><span class="p">;</span> <span class="c1">//count of non spoken words in a subtitle
</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">_nonDialogue</span><span class="p">;</span> <span class="c1">//list of non dialogue words, e.g. (applause)
</span>
<span class="kt">int</span> <span class="n">_wordCount</span><span class="p">;</span> <span class="c1">//number of words in _justDialogue
</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">_word</span><span class="p">;</span> <span class="c1">//list of words in dialogue
</span>
<span class="kt">int</span> <span class="n">_styleTagCount</span><span class="p">;</span> <span class="c1">//count of style tags in a single subtitle
</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">_styleTag</span><span class="p">;</span> <span class="c1">//list of style tags in that subtitle
</span>
</code></pre></div></div>
<h3 id="downloading-the-header-file">Downloading the header file.</h3>
<p>You may download the header file in any way desireable.</p>
<ul>
<li>
<p>Simply clone the repo using</p>
<p>git clone https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp.git</p>
</li>
<li>
<p>Download the zip file from</p>
<p><a href="https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp/archive/master.zip">https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp/archive/master.zip</a></p>
</li>
</ul>
<h3 id="license">License</h3>
<p>srtparser.h library is licensed under MIT License (find it here). Feel free to use it in your application. :) Happy development!</p>
<h3 id="contribution-and-feature-request-bug">Contribution and Feature request/ Bug</h3>
<p>Feel free to raise an issue or make a feature request <a href="https://github.com/saurabhshri/simple-yet-powerful-srt-subtitle-parser-cpp/issues">here</a>.</p>
<p>Also, feel free to contribute to the project. Your help would highly be appreciated! 😀</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc/simple-yet-powerful-single-header-srt-subtitle-parsing-library-in-cpp">Simple yet powerful single header srt subtitle parsing library in cpp</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 29, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc/creating-a-full-blown-srt-subtitle-parser2017-05-16T00:00:00+05:302017-05-16T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Creating a C++ subtitle parsing library to fetch and process subtitle file easily and efficiently.</p>
<blockquote>
<p>EDIT: I have completed building the parser, read more about it here : <a href="https://saurabhshri.github.io/2017/05/gsoc/simple-yet-powerful-single-header-srt-subtitle-parsing-library-in-cpp">https://saurabhshri.github.io/2017/05/gsoc/simple-yet-powerful-single-header-srt-subtitle-parsing-library-in-cpp</a>.</p>
</blockquote>
<p>For my GSoC 2017 project, <a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017">CCAligner - Word by Word Audio Subtitle Synchronization Tool</a>, the very first step required is processing of subtitle to extract primarily two things - the words those are being spoken, and time duration in which they are being spoken.</p>
<p>A usual SubRip (SRT) subtile file has 4 basic components :</p>
<ol>
<li>A number indicating which subtitle it is in the sequence.</li>
<li>The time that the subtitle should appear on the screen, and then disappear.</li>
<li>The subtitle itself.</li>
<li>A blank line indicating the start of a new subtitle.</li>
</ol>
<p>All of these, of course are textual.</p>
<p>E.g.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
1
00:00:00,520 --> 00:00:03,536
Chris Anderson: Elon, hey, welcome back to TED.
2
00:00:03,560 --> 00:00:04,936
<i>(Applause)</i>
3
00:00:04,960 --> 00:00:06,536
Elon Musk: Thanks for having me.
4
00:00:06,560 --> 00:00:09,416
CA: So, in the next half hour or so,
</code></pre></div></div>
<p>Now as you can see, a subtitle file may also include some text in the text field which is not actually <em>spoken</em> but is there to convey some information. The second subtitle is a perfect example of this. The words <code class="highlighter-rouge">(applause)</code> are not spoken but are present in text field. Moreover, styling tags may also be associated as they are valid in SubRip (SRT) format. The <code class="highlighter-rouge"><i> .. </i></code> for example, denotes that the line is to be displayed in italics.</p>
<p>My project (CCAligner), as per plan, should take two files as input - the video/audio file itself and it’s subtitle file. In order to perform alignment, the tool needs to read subtitle file and extract meaningful data in required format. This requires parsing i.e. the tool, needs to <em>parse</em> the subtitle file and present / extract data in suitable format. In order to do so, I am creating a parser which shall perform this task for me.</p>
<p>All in all, the subtitle (SRT) parser must have at least following functionalities :</p>
<ol>
<li>Should be written in C++ (CPP) language, as my tool will primarily be in C++.</li>
<li>Should be capable of returning starting time, ending time and actual subtitle text with simple function calls.</li>
<li>Should be able to extract and strip HTML and other styling tags (e.g. <i> hello </i> to hello).</li>
<li>Should be able to extract and strip speaker names (e.g. Elon: Hi to Hi).</li>
<li>Should be able to extract and strip non dialogue elements (e.g. (applause) to {blank}).</li>
<li>Should be easy to use and efficient in functioning.</li>
</ol>
<p>One of the major advantages of Open-Source is, that one does not need to reinvent the wheel. :) So obviously before head-diving into writing the code I did a bit of search to see what work is already been done. There are not a lot of SRT parser I could find which were written in C++, but I found a one in which there is a base, but everything is very raw and perfect for adding onto. Obviously I wasn’t expecting to find a parser exactly as per my need. <a href="https://github.com/young-developer/subtitle-parser">Here’s</a> the repository.</p>
<p>I have already begun working on it. The original parser could only return starting and ending time (in ms) and subtitle text. I have modified it to be capable of performing the functionalities I listed above. The work is still raw, but I am sure it should come out great.</p>
<p>Right now my srt parser successfully does the following :</p>
<ol>
<li>Return timestamps in both string format and in ms.</li>
<li>Capable of strippin style tags, non dialogue data, speaker names.</li>
<li>Extract speaker names.</li>
</ol>
<p>But there are some issues, after all I have only just begun working on it. These issues are expected to be resolved soon.</p>
<ol>
<li>Style tags and non dialogue texts are stripped but not stored.</li>
<li>Only single-name speaker name is extracted, i.e. works for <code class="highlighter-rouge">Elon: Hi</code> but only extracts last name for <code class="highlighter-rouge">Elon Musk: Hi</code>. I’ll add a check to extract both.</li>
<li>To convert time to ms, I am currently usin regex, which I do not like. Plus it os only available in C++ 11 and above. So, I have to write an alternative for that.</li>
</ol>
<p>I am also planning to make this parser a single header library. After solving the above issues I’ll work on that and upload it as an entirely new repository.</p>
<h1 id="current-parse-output-">Current Parse Output :</h1>
<p>I have implemented a lot of functions in my SRT parser,here’s a demo of some of them :</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">for</span><span class="p">(</span><span class="n">SubtitleItem</span> <span class="o">*</span> <span class="n">element</span> <span class="o">:</span> <span class="n">sub</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"start : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getStartTime</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"end : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getEndTime</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"text : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getText</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"justDialogue : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getDialogue</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"speakerCount : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getSpeakerCount</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">element</span><span class="o">-></span><span class="n">getSpeakerCount</span><span class="p">())</span>
<span class="p">{</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">name</span> <span class="o">=</span> <span class="n">element</span><span class="o">-></span><span class="n">getSpeakerNames</span><span class="p">();</span>
<span class="k">for</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">display</span> <span class="o">:</span> <span class="n">name</span><span class="p">)</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"speakers : "</span><span class="o"><<</span><span class="n">display</span><span class="o"><<</span><span class="s">", "</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"ignore : "</span><span class="o"><<</span><span class="n">element</span><span class="o">-></span><span class="n">getIgnoreStatus</span><span class="p">()</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="n">myfile</span><span class="o"><<</span><span class="s">"____________________________________________"</span><span class="o"><<</span><span class="n">endl</span><span class="o"><<</span><span class="n">endl</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>For the input :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1
00:00:00,520 --> 00:00:03,536
Chris Anderson:
Elon, hey, welcome back to TED.
2
00:00:03,560 --> 00:00:04,936
It's great to have you here.
3
00:00:04,960 --> 00:00:06,536
Elon Musk: Thanks for having me.
4
00:00:06,560 --> 00:00:09,416
CA: So, in the next half hour or so,
</code></pre></div></div>
<p>I recieved the following output :</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>start : 520
end : 3536
text :
Chris Anderson:
Elon, hey, welcome back to TED.
justDialogue :
Chris
Elon, hey, welcome back to TED.
speakerCount : 1
speakers : Anderson,
ignore : 0
____________________________________________
start : 3560
end : 4936
text :
It's great to have you here.
justDialogue :
It's great to have you here.
speakerCount : 0
ignore : 0
____________________________________________
start : 4960
end : 6536
text :
Elon Musk: Thanks for having me.
justDialogue :
Elon Thanks for having me.
speakerCount : 1
speakers : Musk,
ignore : 0
____________________________________________
start : 6560
end : 9416
text :
CA: So, in the next half hour or so,
justDialogue :
So, in the next half hour or so,
speakerCount : 1
speakers : CA,
ignore : 0
____________________________________________
</code></pre></div></div>
<p>Looks good, right? :)</p>
<p>Of course there’s a lot of work left. I will also spend quite a time in optimizing the performance and documenting it so that other people can also use it.</p>
<p>If you guys have any suggestions for the parser, or would like to request a feature, feel free to post in the comments or mail me. I will be adding link to the repository as soon as I upload it.</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc/creating-a-full-blown-srt-subtitle-parser">Creating a full blown (SRT) Subtitle Parser </a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 16, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc-how/here-s-why-i-waited-3-days-before-revealing-about-my-gsoc-selection2017-05-08T00:00:00+05:302017-05-08T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>I got accepted into Google Summer of Code 2017, the result of which was announced on May 4th. I did not tell anyone about this until yesterday, here’s why!</p>
<p>While I had been preparing for GSoC since November, no one in my vicinity knew about it. It wasn’t a very big secret and I had no major reason for hiding it - except for the fact I did not want to get disctracted by the thoughts of what my friends and peers were thinking about it and the constant questioning that would have risen. Working quitely gave me the benefit of remaining focussed and working without any burden of people’s opinions and expectations. It also gave me an amazing opprtunity to surprise my family and closed ones when I finally got accepted into GSoC. I will definitely write about that and update this post with link to it. So visit again some time later if you are interested in reading that.</p>
<p>The thing is, when I decided to try for GSoC (it was around November previous year), I made a promise to myself that my mother would be the first person I tell about this if I get accepted. The problem was, when the result was announced, I was amidst my semester examinations. I sure did not want to miss my mother’s expression by telling her over the phone, I wanted to be <em>there</em> when I tell her this. So, even being the blabber-mouth I am, I somehow had to keep this news to myself.</p>
<p>When the result was announced, I rushed out of room with my laptop and cell-phone to another college building, because there were lot of people in my room at that time. I sat there and logged into the dashboard to find this amazing news. :) I immediately had the conversation regarding the same with the community and then went back to my room to prepare for next exam.</p>
<p>My exams finally ended on 6th May, and I took the morning train to home the immediate next day! I had arranged a surprise announcement for my family and friends and then I finally told my parents about my selection. I would like to thank everyone for their wishes and compliments and I am really sorry to everyone because I did not tell you all earlier about this. But I hope you understand the reason now.</p>
<p>I finally published the long-due post about me getting accepted into GSoC prior to this post (read that <a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017" title="Getting accepted into GSoC.">here</a>) and now that I am home, it’s time to get to work as per my proposed timeline! :)</p>
<p>Thank you for reading, feel free to comment with your views and questions, if any!</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc-how/here-s-why-i-waited-3-days-before-revealing-about-my-gsoc-selection">Here's why I waited 3 days before revealing about my GSoC selection. </a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 08, 2017.</p>https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-20172017-05-08T00:00:00+05:302017-05-08T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>My proposal <em>CCAligner - Word by Word Subtitle Synchronization</em> with CCExtractor Development has been accepted for Google Summer of Code (GSoC) 2017!</p>
<blockquote>
<p>If you are reading this, chances are, you already know what’s GSoC. In case you don’t, here’s official link to the same : <a href="http://g.co/gsoc">http://g.co/gsoc</a> .</p>
</blockquote>
<p>I am very happy to announce that my proposal to build a tool for word by word audio subtitle synchronization has been selected for Google Summer of Code. I will be working with the organization <a href="https://ccextractor.org" title="CCExtractor Website.">CCExtractor Development</a> which made the de-facto subtitle extraction tool - <a href="https://github.com/CCExtractor/ccextractor" title="CCExtractor on Github.">CCExtractor</a>. I am super excited to work with my mentors <a href="https://github.com/cfsmp3" title="Carlos' Github profile.">Carlos Fernandez Sanz</a> (who originally built CCExtractor) and <a href="https://github.com/AlexBratosin2001" title="Alex's Github profile.">Alex Bratosin</a> (CCExtractor GCI 2016 Winner).</p>
<h3 id="what-is-my-project-about">What is my project about?</h3>
<p>I have named my project <strong>CCAligner</strong> as it conveniently lays out it’s basic functionality and also adheres to the name of it’s parent tool CCExtractor. So, what generally happens is that the usual subtitle files (such as SubRips) have line by line synchronization in them i.e. the subtitles containing the dialogue appear when the person starts talking and disappears when the dialogue finishes. This continues for the whole video. For example :</p>
<p>1274<br />
01:55:48,484 – 01:55:50,860<br />
The Force is strong with this one</p>
<p>In the above example, the dialogue #1274 - <em>The Force is strong with this one</em> appears at <code class="highlighter-rouge">1:55:48</code> remains in the screen for two seconds and disappears at <code class="highlighter-rouge">1:55:50</code>.</p>
<p>The aim of the project is to tag the word as it is spoken, similar to that in karaoke systems.</p>
<p>E.g.</p>
<p>The———[6948484:6948500]<br />
Force——[6948501:6948633]<br />
is————[6948634:6948710]<br />
strong—–[6948711:6949999]<br />
with——–[6949100:6949313]</p>
<p>In the above example each word from subtitle is tagged with beginning and ending timestamps based on audio.</p>
<h3 id="why-is-this-useful">Why is this useful?</h3>
<p>While watching a video, it makes sense to have a whole / part of sentence displayed on screen rather than individual words as they are spoken. But there are cases where having timing information of each word is very important. Think of a scenario where you have to tag an occurrence of an event, marked by a special word, then having the information about when the word was spoken is what we need. This is a very basic example to just give you an idea. I have written about various possible applications of this tool in my proposal, and do give it a read if you are interested.</p>
<p>I really hope by the end of summer, the tool gets ready to be used. The basic flow of usage would be really simple. Just call the tool, pass the audio file, the subtitle file, choose the mode, the output type and the result should be word by word subtitle synchronization.</p>
<h3 id="what-am-i-doing-right-now">What am I doing right now?</h3>
<p>Right now, it’s community bonding period. I just finished my exams and returned home. As mentioned in the timeline, I will be spending this month fine-tuning my deliverables by discussing with my mentors, and also making a sample repository for me to test the tool.</p>
<h3 id="the-gsoc-result-was-announced-on-4th-why-such-a-late-post-am-i-lazy">The GSoC result was announced on 4th, why such a late post? Am I lazy?</h3>
<p>While I might be lazy, which I most certainly am, it has nothing to do with me being late. I fell in love with open-source ever since I made my first contribution, and I extremely excited for this GSoC. I was amidst my semester examinations when the result was announced. I had already intimidated my mentors about the same and they themselves advised me to prioritize exams. I will be publishing few more posts about the same later on.</p>
<h3 id="wheres-the-proof-of-selection-and-my-proposal">Where’s the proof of selection and my proposal?</h3>
<p>I am including this, only to show-off my name on GSoC website :P .
This is the official link about my project on Google Summer of Code website - <a href="https://summerofcode.withgoogle.com/projects/#5589068587991040" title="CCAligner - Word by Word Subtitle Synchronization | Google Summer of Code Project by Saurabh Shrivastava">https://summerofcode.withgoogle.com/projects/#5589068587991040</a> . I shall also soon add myself on CCExtractor’s website.</p>
<p><img src="/images/posts/accepted_proof.PNG" alt="My Project listed on GSoC website." /></p>
<p>About my proposal, you may find my GSoC 2017 proposal for CCExtractor Development <a href="https://drive.google.com/file/d/0B7xAF1f7vzYzOE1LY21pZjdSTW8/view?usp=sharing" title="GSoC Proposal - CCAligner">here</a> and in case that link doesn’t work (please comment about the same, I shall replace it), here’s the <a href="https://github.com/saurabhshri/saurabhshri.github.io/blob/master/GSoC/5565268630700032_1490805743_Word_by_Word_Subtitle_Sync_by_Saurabh_Shrivastava_CCExtractor.pdf" title="Mirror of my GSoC proposal on Github.">mirror</a>.</p>
<p>Follow this blog to read my future posts. Thank you for reading. Feel free to comment with your views, questions and criticism, if any. I would love to discuss them. :)</p>
<p><a href="https://saurabhshri.github.io/2017/05/gsoc/accepted-in-google-summer-of-code-2017">Accepted in Google Summer of Code 2017!</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on May 08, 2017.</p>https://saurabhshri.github.io/2017/03/personal/news/hello-world2017-03-11T00:00:00+05:302017-03-11T00:00:00+05:30Saurabh Shrivastavahttps://saurabhshri.github.iosaurabh.shrivastava54@gmail.com
<p>Hello everybody! This blog will contain all the updates by me. I will try my best to update it regularly. Stay tuned!</p>
<p>I am Saurabh Shrivastava a 3rd year Information Technology Engineering undergrad at <a href="http://ietdavv.edu.in/">IET DAVV, Indore</a>, India. My interests lie in the field of computers. I was introduced to programming in 11th standard and scored a perfect 100 in my 12th Computer Science Board Examinations. I love programming in C and C++ and am constantly learning new things.</p>
<p>I recently got introduced to the world of open source and I haven’t looked back since. I absolutely love the idea of open source software and am thouroughly enjoying it. I am trying my part to <a href="https://github.com/saurabhshri" title="My Github page.">contribute</a> as I learn :) It feels amazing to know that your contribution is <em>actually</em> out there and people are using it. I also started competitive coding at <a href="https://www.codechef.com/users/shubhshri" title="My CodeChef Account.">CodeChef</a> but haven’t been active since sometime. I shall start again as and when I get time (should be soon after this semester ends).</p>
<p>You may read more about me in the <strong>About Me</strong> section which I should be updating soon.</p>
<p>This blog post is basically to test this newly set up blog. You may read more about the details of blog set-up below :</p>
<ul>
<li>
<p><strong>Platform</strong>: The blog is hosted on <a href="https://pages.github.com/">Github Pages</a> with the help of <a href="https://jekyllrb.com/">Jekyll</a>. The blog was up in literally minutes without much hassle.</p>
</li>
<li>
<p><strong>Theme</strong>: The theme is <a href="https://github.com/hmfaysal/Notepad">Notepad</a> by <a href="https://twitter.com/hmfaysal">@hmfaysal</a>. I have always been a fan of plain and simple design.</p>
</li>
<li>
<p><strong>Source</strong>: You may find the source of this blog <a href="github.com/saurabhshri/saurabhshri.github.io">here</a>.</p>
</li>
</ul>
<p>Thank you for reading! :) Comment here to share your suggestions.</p>
<p><a href="https://saurabhshri.github.io/2017/03/personal/news/hello-world">Hello World!</a> was originally published by Saurabh Shrivastava at <a href="https://saurabhshri.github.io">//Saurabh Shrivastava</a> on March 11, 2017.</p>