Posted in 2017, LearningItMyWay, Lightbeam, Outreachy, Random

Lightbeam updates

Here are two quick updates about Lightbeam because I can’t contain the excitement to myself.

Lightbeam goes responsive, yayyyyy!!!

lb.gif
Responsive Lightbeam

Responsive UI

I am extremely happy for achieving this today. Making the UI responsive was there in our to-dos, but this one got done today accidentally in an attempt to answer few of the comments on one of my PR. CSS grid is used and I must say the fr unit is so handy.

Mozfest proposal submission

Proposed a session for the Mozilla Festival 2017, London. Here is the proposal. The coolest part here is that the google-form submission automatically gets created as a GH issue.

Advertisements
Posted in 2017, JavaScript, LearningItMyWay, Lightbeam, Outreachy

Lightbeam – SVG to Canvas

SVG is the preferred choice for D3. But when you expect a lot of DOM nodes (yes, in the case of Lightbeam) you need to start worrying about the DOM performance and have to take the call to step out of the SVG comfort zone. We chose Canvas over SVG for Lightbeam 2.0!

In this post, I would like to highlight the key points of drawing and interactivity on the HTML5 Canvas element.

d3andcanvas
Link

 

In our case, we followed the first approach – using D3 solely for its functional purpose (D3’s force layout algorithm) and then drawing onto the canvas.

D3 joins

Normally, when you follow this approach, you are trading off D3’s super rich data binding and joining functionality. It means you’re only drawing your graph once – you’re not expecting new data that would require a graph redraw or update. This is why D3 with some dummy HTML nodes is a preferred way in order to retain the data binds and joins. D3’s joins are a way to dynamically update the graph without having to redraw the whole thing over and over again.

Screen Shot 2017-07-28 at 11.51.35
Typical example of enter, update and exit methods in D3 joins. Link

The dummy HTML element approach

Here is an example of using custom aka dummy HTML elements to render D3 on canvas.

Screen Shot 2017-07-28 at 11.59.52
custom elements

custom is definitely not a standard DOM element type, so it will not be rendered in any way and will live only in memory (virtual DOM). It is used as a container for other dummy nodes.

How does Lightbeam update dynamically without D3 joins?

Even though we followed the first approach, Lightbeam has dynamic updates. Luckily, D3’s force layout algorithm takes care of the new node and link updates/additions and we managed to use D3 only for its functionality without the dummy element approach. Here is the PR.

Interactivity on the canvas

Canvas is a single DOM element and mouse interactions on the canvas can be sometimes tricky because you don’t have independent access to the nodes and links (or any graph element). In our case, we have the following interactions:

  • On hover over the nodes (websites) show tooltips with the name of the website
  • Drag the graph
  • Zoom in and zoom out the graph
  • Panning – shifting the view of the graph

At the time of writing this blog post, only tooltip based interactivity is achieved. PR

I shall explain tooltip based canvas interactivity in the next post.

Resources:

Initial performance results

I have never done performance testing using browser devtools. When we migrated from SVG to Canvas I was curious to know the performance results of using canvas. For a given small sample of websites, here are the test results. Canvas has a rendering score of 4.3 milliseconds 🙂

old
Old Lightbeam (SVG)
new-svg
New Lightbeam (SVG)
new-canvas
New Lightbeam (Canvas)
Posted in 2017, JavaScript, LearningItMyWay, Lightbeam, Outreachy

Lightbeam – Tooltips (SVG)

In this blog post I want to share my lessons learned from creating tooltips for the Lightbeam visualisations.

tooltip-gif
Tooltips – canvas based solution

Visualisations are the major part of this Outreachy – Lightbeam internship. We use D3-force to create a simulation for an array of nodes (websites), compose the desired forces, and then listen for tick events to render the nodes as they update. When we started, the preferred graphics renderer was SVG. We now have a Canvas based implementation because this is more efficient than SVG. Here is an article why canvas is better than svg and here are the initial results.

Having said that, lets talk about tooltips.

TL;DR

SVG based solution

I had first implemented tooltips(text labels) on the SVG version. Here is the PR.

tooltips-svg
Text labels – svg based solution

In SVG, you can render every node as a DOM element. This is why SVG isn’t the right solution for graphs with too many nodes, because it can easily slow down the browser’s page loading time due to too many nodes.

Since this is an SVG based solution, I chose text-labels over tooltips.

Tooltips vs text-labels

Tooltips come with an additional overhead of dynamically updating the x,y coordinates and the title. Text labels are rendered like ordinary DOM elements, the SVG <text> element, one for each node(website).

But using tooltips, you can cut down the number of text nodes. As opposed to having one text node for each node(website), there is one tooltip (DOM element) whose position and content is dynamically updated.

I shall discuss the tooltip based solution in the canvas implementation.

Implementation

1) Text labels in SVG

  • By default, the visibility of text-labels is set to hidden.
  • All text-labels have a common class textLabel.
  • On hover, over the circular nodes, the corresponding text-label’s visibility is changed from hidden to visible.
  • For each node(website) there is a corresponding <text> element.
  • In order to identify and set the visibility of the right <text> corresponding to the hover on node(website), every <text> element is assigned an additional class text-i where i is the index corresponding to each node(website).
  • When node(website) with index i is hovered, the corresponding <text> with class text-i is set to visible.

drawlabel

2) Hover on the nodes(websites)

In order to achieve the hover effect, mouseenter and mouseleave event handlers are registered on each node(website).

drawnodes

Fixing Bugs

hover-flicker

After the initial implementation, there was a lot of flicker when the text labels were made visible on node hover. Although it isn’t very clear from the above gif, notice the time taken to display http://www.google.com on the third node. Looks like gifs aren’t good at capturing flickers (I need to find out a better solution), but the time delay to show the third label in the above gif === flicker effect.

1) mouseenter vs mouseover

At first, I thought the flickering was due to the mouse events. Later, I found out that these mouse events had nothing to do here. However, it is worth making a note of the above two mouse events. Link

mouseenter Event reference MDN

Below is the DOM for the SVG version:

Screen Shot 2017-07-21 at 16.55.12

Since the nodes/circles are independent of the text labels, and have no direct descendant nodes, the mouse events had nothing to do with the flicker.

2) SVG and z-index

After wracking my brains for a while, I figured out the cause for the flickering of labels. If you carefully examine the above gif, you will notice that the lines are on top of the circles. On mouse hover over the nodes, it was mouseenter for circles and labels were made visible. But, when the mouse accidentally appeared on top of the lines (because you don’t realise you are doing so) it caused the mouseleave for circles and the visibility was getting toggled.

When I figured out this cause, the obvious solution to run in my head was to set z-index for circles to be greater than the lines. Despite this, the flickering was still there. On further investigation, this is what I found from the SVG specification:

Screen Shot 2017-07-21 at 17.16.24

z-index in SVG is defined by the order the elements appear in the document. You will have to change the element order if you want to bring a specific shape to the top. Drawing lines first, followed by drawing circles fixed the flickering bug.

It was good to learn about z-index in SVGs.

To be continued… tooltips – a canvas base implementation

Posted in 2017, Outreachy, Random

Am I a fraud?

Imposter Syndrome is described as a feeling of “phoniness in people who believe that they are not intelligent, capable or creative despite evidence of high achievement”. While these people “are highly motivated to achieve”, they also live in fear of being ‘found out’ or exposed as frauds. (Clance and Imes, 1978)

Recently, I had this wonderful opportunity to participate in the Mozilla All Hands, 2017. There was an Imposter Syndrome workshop for Outreachy participants and here are the session highlights. The workshop was led by Lizz Noonan from the Diversity and Inclusion team, and we learned techniques to identify and overcome the Imposter Syndrome.

We were aproximately 12 participants and Lizz started the session with an introduction to the Imposter Syndrome. We were then asked to introduce ourselves and also to state ‘one thing you didn’t know looking at me‘! In my case, “I turned 30” 😉

Imposter Syndrome is the belief or the feeling that you are a fraud! It often starts with ‘I can’t do this‘ or ‘Can I do this‘?! Lizz cited examples from noted women – Maria Klawe, President of Harvey Mudd College, Cherry Murray, former Dean of the Harvard School of Engineering and Applied Sciences, stating how they too have been bitten by this bug.

Imposter Syndrome disproportionally affects women, so it is something of which we should be especially aware. This wasn’t the first time that I was hearing about Imposter Syndrome; I have heard and read articles about this in the past. What I didn’t realise until this workshop is that I too have been silently affected by this bug.

After a brief explanation about this syndrome, we were asked to share our experiences. When it was my turn to share my Imposter Syndrome experience, I realised why I have always (and still I am) been reluctant to asking questions in public?! There have been numerous times when I have had framed a question in mind, and then wondered to myself that this could be the most stupid question to be asked, felt shy to pop it out of my head, and then hear someone else ask the exact (or similar) question and get applauded for asking the best question. Sigh! This still hasn’t boosted confidence in me to ask a question the next time. At a technical workshop, slack discussion, PR comments, I feel too naive to ask any question.

This workshop made me realise that this behaviour of mine is because of the fear in me of being exposed as a ‘fraud‘.

Why do we do this to ourselves?

  • You get the feeling of an impostor when your competence is questioned all the time – “Did you get into this program because you are a woman?
  • Often, you feel like an imposter because people treat you like one.
  • People act surprised when you are good.
  • There is this ‘prove -it-again‘ bias.
  • And ‘lower your ambitions‘ thing.

What is the result from the above?

  • We waste time over-preparing
    • In my case, instead of asking a question, I go and dig every possible source to help me find the answer myself.
  • We settle for less money (salary).

How do we prevent this?

  • Share your failures
  • Doc doc doc (doc = document, you don’t need to see a doctor :))
  • Tell your story (This is my story)

After sharing our experiences, our next exercise was to ‘Take a Compliment” and say “Thank You” for the compliment. We were divided into groups to exchange compliments and thank you notes. We often receive compliments but the imposter in us stops us from saying a thank you.

Lizz shared her own experience of making a log of all the thank you notes she receives and how it helps her when she is low.

Thank you Lizz for this great workshop! I have spoken to my Outreachy mentor Stephanie Ouillon about my reluctancy in asking questions and she has come up with a nice way to help me be confident in asking questions. So the plan is whenever I am not sure to ask a question during our weekly team meetings, I first shoot the question to her and get the boost to ask it to the whole team. We have our weekly meetings every Tuesday, today is Sunday, let’s hope for the best 😉

Posted in 2017, Lightbeam, Outreachy, Random

Lightbeam – Behind the scenes

It’s more than 2 weeks now. I haven’t had the time to write anything down. So here’s a quick post on what’s happening!

San Fransisco – Mozilla All Hands 2017

Before

The week of June 26 – 30 was Mozilla All Hands 2017 at San Fransisco. It was my first time into the United States and I was pretty excited about this trip. After a tiring 11 + 3 hour long haul flight, when I presented my Mozilla invitation letter at the immigration, the border security officer at the SFO airport was excited to find out that I too was attending the All Hands. It appeared to me that he already processed other fellow attendees and he knew about it. His cheerfulness made me forget my tiredness and I had a warm welcome into the city. With the kind of ongoing news and rules about US airport security, I was too paranoid. But everything went smooth, from gaining B1 visa, carrying laptop in hand baggage and the security clearance itself.

During

The All Hands itself was a great experience. I and Bianca came up with a timeline of the next tasks that we could work on for until the internship period. Our main focus is to improve the graph performance. The idea is to use canvas instead of SVG. Here is an article I have written why canvas is a better choice.

After

I did a bit of sight-seeing at SFO via the big-bus tour.

One week at SFO was great. I wish to go back and explore more of the city at some other time.

Berlin – Home, away from home

Store

store is an integral component of Lightbeam. We use asynchronous message passing for communications between the background and page scripts. Our mentor Jonathan helped us a lot with code improvements on this store.

Visualisations

28039210-b72434a8-65c1-11e7-932f-63a725a4438e
Lightbeam graph using D3 force layout
28042023-6cee9a36-6580-11e7-8de5-5089702ae68c
Node overlap/colliding bug

I have used D3 in the past, but D3’s force layout algorithm is new to me. We are able to draw the Lightbeam graph using this algorithm, but there are so many more things to learn. I am sure, over the coming weeks our graph would see significant improvements.

This week I am working on to fix the node overlap/colliding issue and to get a canvas prototype for D3’s force layout.

Quite exciting times with Lightbeam 🙂

Posted in 2017, Lightbeam, Outreachy

Lightbeam – ‘tale of 2 stores’

Thanks to Bianca, my fellow Outreachy-Lightbeam project partner for coining the above title in her blog and asking me questions to explain about the bug(PR) I recently fixed for Lightbeam. I  started explaining her last evening via quick diagrams and I already had an idea to make my next blog post colourful 😉

I have been inspired by Lin Clark’s code cartoons and Mariko Kosaka who always draw and make technical reading fun! Here is an attempt to write a post and explain Lightbeam’s store architecture via diagrams 🙂

To revamp Lightbeam into a web extension we decided to follow the following architecture:

20170623_125927

A web extension consists of a collection of files and we employed the following directory structure to achieve the above architecture:

20170623_125810

We were able to chalk out an MVP soon, everything behaved as expected, until we found this bug.

  • Open Lightbeam
  • Open console
  • Clear the storage manually through the console: browser.storage.local.remove('websites').then(()=>console.log('removed')).catch(err=>console.log(err))
  • Also set store._websites=null manually through the console
  • Refresh the page – graph disappears
  • Refresh the page again, graph appears with old values

Thanks to Jonathan for pointing out that there were two store instances created and thus began – ‘tale of 2 stores’.

Following was the code implementation to achieve the above architecture:

The problem with this approach was that store.js file was loaded from two different places and that’s how we ended up having two store instances.

In other words, this was the problem:

20170623_130426

Now to debug the above specified bug:

  • capture.js had one store instance because of store.js loaded via background script
  • lightbeam.js had another store instance because of store.js loaded via the script tag
  • index.html page was loaded and lightbeam.js got the required websites to display via getAll() from it’s store instance
  • Next, web extension storage was manually cleared via browser.storage.local.remove and store._websites was explicitly set to null
    • in the first page load, there was nothing to display because the getAll() returned nothing from it’s store instance (because of the above manual deletion)
    • on second page load, capture.js had done a setFirstParty() call to it’s store instance which in turn wrote to the browser.storage.local.set
      • capture.js is triggered when a tab (re)loads
      • capture.js‘s store instance had its copy of _websites already populated (when the manual store._websites=null was done, it was done on lightbeam’s store instance) and setFirstParty() used this already populated _websites value and wrote to the browser.storage.local.set
  • This is how the visualisations appeared back on the second page load

To fix this bug, we now have storeChild.js which acts as an interface, or is the only point to talk to the background store.js. With the child and parent architecture, the parent will always be in charge of when the write happens and so will serialise the writes which will behave like a locking database.

20170623_130453

The page script storeChild.js talks to the background script store.js via message passing.

Posted in 2017, Lightbeam, Outreachy

Lightbeam 2.0 – MVP

It’s been 4 weeks now, since we started working on Lightbeam 2.0. And we have an MVP!

Here are my earlier posts on Outreachy and Lightbeam:

The first milestone in this internship is this initial MVP. Every time, I and Bianca went astray from this milestone, we had our mentor Jonathan to guide and help us focus towards this direction.

Here is the summary of what’s been achieved so far:

Setup the basic web extension

lightbeam-setup-initial

The Backend

  • Setup the communication with the browser and capture the first & third party websites
  • Create the capture & viz objects and the store API
    • capture object stores into the store object
    • viz object draws to a canvas
    • The lightbeam page code reads from the store, draws with the viz to a canvas in index.html

The Frontend

  • UI
    • In this initial MVP, the goal was to replicate the old UI. CSS Grid layout is used in the new codebase
  • Visualisations
    • While visualisations will be a major part in the post-MVP, in this initial MVP we are drawing small circles using the Canvas API to show the first and the third party websites

Overall, the whole team is very happy with the work and the progress done so far.

Presenting, Lightbeam 2.0

Screen Shot 2017-06-18 at 18.46.11