Ana Balica

Right time, right place

2020-10-15T00:00:00+00:00

I started reading Outliers by Malcolm Gladwell. Believe it or not, this is the first book I picked up by him.

I’ve been pleasantly surprised how entertaining the book themes are - everybody likes a good story tied in with social science research. For me personally though, the first few chapters clarified why I feel so lucky.

Even since I saw this talk by Saron Yitbarek, I tried to understand what made me so fortunate. Because, well, I feel pretty fortunate. It’s a good life phylosophy, but the issue is that I believed that every single event and circumstance was a blessing (in retrospect, of course). Now I can single out one aspect that is more important than others.

I was born in 1990 in Moldova, as the Soviet Union was about to collapse. By the time I was in my early 20s, the borders were open enough to travel, study and live abroad.

Would I have been born too early (I’m thinking 20-30 years earlier), it might have been challenging to part ways with the old world and build a new life. I probably wouldn’t have had the same resources or need to learn foreign languages. I wouldn’t have traveled to Europe for fun. I wouldn’t have stumbled upon software engineering or had the opportunity to pursue a fruitful career in this domain. If I were to be born later, no idea what would have happened, as the history is still unfolding.

I’m not an outlier, I’m ok with a local maxima. But given the place, I showed up at a pretty good time.

I know that ‘60-70s would have been a perfect time for some, but given my business ignorance, I probably would have missed the opportunity.

How to feedback

2020-10-06T00:00:00+00:00

These are some things I learned over the years about feedback.

Why

Never loose focus on why we give and receive feedback. We do that because we want to help others excel and to better ourselves. Further below I break it down into 2 parts: giving via Situation-Behaviour-Impact model and receiving via Listen-Explore-Echo (not a coined term, sort of my interpretation).

Situation-Behaviour-Impact

It only works if the feedback is heard and understood by the receiver. So it matters what you say and how you deliver it. I like SBI, because it’s simple, effective and it reduces bias.

SBI breaks down the feedback into 3 components:

Situation - this is something that happened at a certain time and at a specific location. Normally both the giver and the receiver can agree on it as a fact. Example: “During the team meeting last Monday we were discussing the project X plan for the next quarter.”
Behaviour - describe the behaviour that you’ve observed. What did the other person do? Avoid your personal interpretations and don’t try to guess their motivation. Example: “When person Y was talking about a design approach, you interrupted them mid-sentence.”
Impact - explain how did the behaviour in that particular situation affected you, the team, the organisation. It doesn’t have to be quantifiable impact, you can talk about your emotional state in that situation. Optionally you can offer a solution. Example: “I felt that you didn’t give them a chance to fully express their idea and that might impair their desire to share with the team in the future.”

Listen-Explore-Echo

Giving feedback is damn challenging, receiving - even more so. You have to suppress the natural fight-or-flight response and not get instantly angry or defensive.

This is my made-up feedback response model:

Listen - don’t interrupt or start making excuses, give space and time for the other person to share their thoughts. If you can, take notes (albeit might be rude in some situations), because in the current state of mind it’s hard to keep track of everything. Your mind will want to cling to one particular thing and it will loose focus. This is a reason why people often prefer written down feedback, as they can take their time to process it.
Explore - don’t assume, ask. Feedback comes in all shapes and sizes, so sometimes you need to poke at it a little bit more. Your end goal is to understand what the other person is trying to tell you, so go ahead and ask questions.
Echo - finally repeat back the feedback and ask if you got it right. This solidifies your mutual understanding of the situation, helps avoid any misinterpretations and sends a positive signal to the other person that they are being heard. If you can, you might start discussing solutions.

Closing thoughts

These models are just guidance. Whether it’s face to face (which is my personal preference) or in written form, you can adapt and shape these models accordingly. Do you have a different approach? Let me know!

Remember, feedback is hard. Change is hard. You’re doing your best. Keep at it!

Shameless reading of notes on stage

2018-09-30T00:00:00+00:00

I’ve given talks on stage since 2015. With time and practice I found my own style and pace. I learned things that come to me easily and things that I struggle with.

One of these things is speaking fluently and not making mistakes, which is tricky for someone like me who is not a native English speaker.

I remember preparing for my first talk and discovering speakers view. I was genuinely astonished to have a separate screen just for myself where I can see the timer, the next build step and, most important, my notes.

Pretty early on I decided my focus is the audience, which meant I want to present well-researched material that is meant to be understood (rather than just skimmed over or be a show-off). To accomplish this I usually need good visuals and a good script.

Putting aside the complexities of expressing information with visuals, verbal delivery is essential. Here are some of the reasons why:

keeping the right pace. Usually talks have a limited amount of time. You don’t want to spend extra 3 minutes explaining the part about header compression and then rush over the interesting statistics that you added to your finishing slides.
keeping the right content. It’s not unusual to see yourself diverging from a topic.
keeping it fluent and concise. I could ramble and say something in 3 sentences with a couple of umm, ehh and pauses to remind myself that word I just forgot. Or I could have a well-formed one sentence written in advance that conveys the same information, but with less overhead.
avoid mistakes. Correct grammar and usage of a language will help deliver your message in a non-ambiguous way.
never go silent. If you ever do, notes are there to help you pick up where your mind went blank.
publishing your talk notes. Like I did with Humanising among coders. This is helpful, because it’s indexable, scannable and ideal for people with hearing problems.

If you can do this without any external aid, I applaud you in the most sincere and mildly jealous way. I can’t. That’s why I use a script. I write everything I want to say from start to end. I usually do it slide by slide. The volume of the content helps me understand if my slide is overloaded with information and I need to split it in 2. When rehearsing I can see the total time talking and be confident that it’s gonna be consistent when I get up on stage.

With a script in front of me, I managed to give pretty bad robotic presentations. I got sucked into the strict flow that I’ve set for myself and didn’t manage to inspire the audience.

Meanwhile I’ve given really good and seamless presentations. The trick there is a well rehearsed script, where you keep track mentally of the broader topic and sentiment that you want your audience to perceive. Read it with understanding and emotion. It will bring you great pleasure when a talk is delivered smoothly with thoughtfulness and care, when it has a quality of a story told by a compelling narrator.

You choose what works for you: no notes, single-worded notes per slide, rough topics, partial or full scripts, as long as you feel comfortable to present your ideas on stage. There’s no shame in being well prepared, whatever that takes.

Empathetic bug report

2018-09-08T00:00:00+00:00

“Feature X doesn’t work”.

Sigh, take a deep breath, pause for a bit… start asking questions: what exactly doesn’t work, what do you see on screen, do you have a URL or screenshots, what environment are you on, are you logged in, what permissions your user has, etc. I get it, sometimes it’s hard to know what additional information is relevant, especially for people who have had less experience interacting with developers. But even we (engineers, product owners, designers) make this mistake every so often.

That’s not the biggest issue though. What matters more is phrasing. The assentive tone of the sentence sends a subtle message. It somewhat implies that the feature X doesn’t work regardless of circumstances, for all users and the whole team managed to spend a couple days work and ship nothing.

“Feature X doesn’t work for me”.

By focusing on your personal experience with the app, it’s possible to approach the bug report in a more humble way, admitting that it might be “you”, “your settings”, “your machine” that are causing the app to act in a weird way. At the end of the day, the quite blunt obvious defect is often a cause of misconfiguration, wrong environment, misunderstanding of the feature (which admittedly can be a bug, but should have been caught during early discussions and refinement).

Next time at work when you’re raging over a feature that doesn’t work, don’t forget to add “for me”.

How to filter by DateRange with django-filters

2018-08-12T00:00:00+00:00

Django offers a few fields that are exclusive to Postgres. I recently had to use DateRangeField which is an interesting alternative to doing the usual start_date and end_date.

If you’re using Django Rest Framework, you are probably familiar with django-filters. It’s a generic system for filtering Django QuerySets, though I’ve mostly used it in combination with DRF.

Timestamped

Tested with following versions:

Django 1.11 (confident this will work with Django 2.1)
django-filters 2.0.0 (won’t work with previous versions, because 2.0 release includes a reworked version of RangeWidget)

Requirements

Ranges are composed of lower and upper bounds that operate as a whole value. I want to be able to specify both bounds when filtering and get the objects that satisfy these exact values. By default, Django stores DateRangeField in its canonical form [) - lower inclusive and upper exclusive. In this case, I’m fine to specify and filter the bounds only in their canonical form. But it should be fairly easy to extend this to support different range types.

From the filters list I spot DateRangeFilter and DateFromToRangeFilter. DateRangeFilter defines a list of choices to select from (like today, yesterday past week, etc), not exactly what I’m looking for. DateFromToRangeFilter looks more appealing, as it allows filtering using the _after and _before suffixes. It will conveniently convert the passed strings to DateField and do gte and lte lookups on the field. Still not exactly what I need.

Show me the code

import django_filters
from psycopg2.extras import DateRange


class DateExactRangeWidget(django_filters.widgets.DateRangeWidget):
    """Date widget to help filter by *_start and *_end."""
    suffixes = ['start', 'end']


class DateExactRangeField(django_filters.fields.DateRangeField):
    widget = DateExactRangeWidget

    def compress(self, data_list):
        if data_list:
            start_date, stop_date = data_list
            return DateRange(start_date, stop_date)


class DateExactRangeFilter(django_filters.Filter):
    """
    Filter to be used for Postgres specific Django field - DateRangeField.
    https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/fields/#daterangefield
    """
    field_class = DateExactRangeField

In your app:

import django_filters

from .filters import DateExactRangeFilter
from .models import Event


class EventFilter(django_filters.rest_framework.FilterSet):
    event_dates = DateExactRangeFilter()

    class Meta:
        model = Event
        fields = [
            'event_dates',
        ]

Explanation

Let’s start from the bottom - DateExactRangeFilter. Since we are doing a regular lookup, we are inheriting directly from Filter. The field_class here is important, as it converts the raw values to a format and type we need, in this case DateRange composed of start and end dates. The DateExactRangeWidget lists the available suffixes.

Now you can do this /api/events?event_dates_start=2018-08-07&event_dates_end=2018-08-12 to filter all events that start and end exactly on these dates.

Warning

This solution is not ideal, because none of this documented. Django-filters doesn’t provide a guide on how to extend existing classes to add your own filters, fields or widgets. The details of how these objects behave might change with time, hence use this at your own risk.

My parents told me I'm pretty

2018-07-21T00:00:00+00:00

As I child, I hated my skin colour, my name, my boring hair colour, my height, my asymmetrical eyes and lack of lashes on a patch of my left eye. When I say “hate”, this was the literal word I’d use. Probably I’ve been bitter about lots of other things, but I can’t remember them all now. Largely because I believe my parents eradicated most of these insecurities by constantly reminding me that I’m pretty.

The power lies not in just repeating this like a mantra. It didn’t feel like it was just parent’s blind love - “No matter what others think, we love you anyway”. Nah, wasn’t like that.

Say, I was bummed out about my darker skin. I know I look white to most, but in my community I was slightly darker than others. My mom would always emphasise how beautiful my olive skin looks and how rare it is. She’d include this in every homework description I had to write about myself. She’d point out how other famous people have similar skin tone - this made me pay attention. Now I’m perfectly fine in my own skin (pun intended).

It went on and on, picking one thing at a time and making me proud of just how it is. Interestingly I think this didn’t make me narcissistic or obsessed with beauty over values and brains.

I have no idea if they did this deliberately, but they managed to make me accept who I am.

So, I guess, tell your close ones they are pretty 💛

Dealing with disagreements

2018-07-03T00:00:00+00:00

You are in an interview. The interviewer across the table asks you: ‘What do you do if you disagree with someone on your team? How do you resolve this situation?’. You pause and take 10 seconds to think about it.

The tricky part is that there’s probably no right answer. It depends, right? What does it depend on though?

For me, there’s one core factor: impact of the decision. Sometimes we like to argue for things that are pretty minor and are most probably easy to correct if they go wrong.

Say, you are arguing for solution A and Kathy is convinced we should go for solution B. You have experience with these things, so you are solid that solution A is the right thing to do. But somehow your wisdom and compelling arguments aren’t reaching Kathy. Stop and assess the impact of going for solution B and imagine what would it take to correct the course. You might find out that this correction is pretty minuscule or at least fairly harmless. Go for solution B! Two reasons:

You want them to taste the soup. They need to try and come to their own conclusions on this topic. The learnings from this will have a stronger effect and will be backed up by real experience.
Who knows, maybe you’ll learn something new. It’s useful to introduce a bit of entropy in our lives. Once we are settled in our ways and we “truly see” the world for what it is, we loose any capacity to be flexible and learn.

What if the impact is massive? Everybody in the room agrees we want to land on the best available solution.

Step away from this and ask yourself why do all of us disagree? Assuming I find myself in a group of well-meaning smart people, I can’t help but wonder what am I missing that I find it hard to accept a different opinion. This is quite profound to think about for me, because it shifts the focus from this isolated case centred around this fixed topic to a wider perspective that sheds light on our core concepts, values, rules and ideas that each one of us settled for. I want to make sure the foundation is same for everyone. I want to know now if I missed something important so that I don’t continue diverging to wrong conclusions. These disagreements can uncover gaps in our shared understanding of the problem. Find this problem and fix it.

All in all, these pesky disagreements are useful as they teach us to work with others and help us uncover interpretations.

The Cemetery of Ideas or Why I Stopped Writing

2018-06-03T00:00:00+00:00

My last blogpost was published on November 11th 2017, which would make it over half a year old.

I don’t think the reason for my writing blackout was general busyness of life. Yes, I did transition into a new role and I’ve taken on new challenges, but if I wanted to write something, I would have done so regardless of time constraints.

Truth is I have a backlog of article ideas. I look over them every Sunday during the review of my TODO list. Something stops me from elaborating on any of them and uploading them here.

A recent short discussion made me think should one post for the sake of keeping themselves going or should one filter and invest in writing only when there’s something worth of sharing.

What I struggle with most is that if it’s not something that I have wrestled with, and spend a lot of time on it, or invested myself directly into it, I don’t feel like I have the permission to talk about it.

There are different types of posts. I could write more stuff about Python and Django, very specific technologies that I have experience with, but because I swim in this day by day, I don’t find any of it all that interesting or innovative to talk about. Moreover considering there are so many other excellent resources on the internet, I’m gonna be yet another lost voice repeating the stuff that’s already known.

Then there’s the stuff I have spent more time on, for example my posts about Datastore indexes ([1] & [2]). I honestly don’t think anyone will ever read this, because it’s such a niche thing. But because I paved my own path to understanding and figuring out some tips and tricks by myself, I feel like I’m genuine when I talk about it.

I thought I want to write about the nature of bugs in software. Working titles: “Embrace your bugs”, “We all have bugs”, “Bugs: let them be”. I can trace the idea back to this book: Turn the ship around by L. David Marquet.

I’ll give you the gist: strive for excellence, don’t focus on avoiding mistakes. If mistakes is all you care about, dodging bugs will drain all you time and energy. And bugs happen all the time. See what lies beyond and aim for excellence.

I read the book by the recommendation of my manager. This idea from the book is not original. But because I know the source of the idea and the way I came to is certainly not original, I grapple taking the bugs blogpost any further.

Quite often when I write, I feel not like an imposter, but like a thief. I read, listen, compile, reshape, put it in a pretty box and present it to others.

This is why I haven’t been writing.

Everything is a remix, but one hopes to create a remix that will become the original for other remixes.

Python: mutable vs immutable

2017-11-11T00:00:00+00:00

Mutable means an object can be modified after it’s been created. In contrast immutable is an object whose state can’t be changed after its creation. Python has both mutable and immutable data types. There’s a simple recipe to figure out if an object can be “mutated”:

initialise a variable of the target type
store the ID (Python has a build-in id() function. In CPython this is the address of the object in memory.)
mutate the variable
check the previously stored ID vs the current ID of the variable

Mutable types

Take lists.

>>> salad = ['spinach', 'tomatoes']
>>> id_before = id(salad)
>>> salad += ['eggs']
>>> id_after = id(salad)
>>>
>>> id_before == id_after
True

I created my original salad with 2 ingredients. Then I added another ingredient - eggs. Even after the mutation (adding eggs), the salad object is still pointing to the same place in memory, it has the same ID.

Even though we have used the + to mutate the list, Python exposes many methods that imply that lists are mutable: list.append, list.pop, list.remove. With its API design programming languages usually try to tell us what is mutable and what is not.

Immutable types

Take strings.

>>> my_string = 'foo'
>>> id_before = id(my_string)
>>> my_string += 'bar'
>>> id_after = id(my_string)
>>>
>>> id_before == id_after
False

Here we are doing something similar to what we did with the salad list - use + to append to the original string. However in this case the object after the mutation happens to reside in a different place in memory.

In fact, Python doesn’t really provide any methods to modify a string “in place” (or give us the illusion we can do that), hinting on the fact that strings are immutable.

Now you’re able to understand better how mutability works. Have fun figuring out the mutability of the other Python data types!

Struggling for creativity

2017-09-19T00:00:00+00:00

When I was in middle school (somewhere between 12 and 16 years old), I wrote an essay for our English class about the butterfly life cycle. It was a short read and in some way beautiful (for a school assignment). I was proud of it. I remember my classmate asked me how did I manage to write this. I shrugged it off, saying I just did it.

Complete lie. I had a very meticulous process that I followed when writing anything.

It’s all about the slow start and building blocks. As kids we first learn words to build up enough vocabulary. From there we are asked to create sentences. This one always puzzled me, because it’s tricky to come up with a good sentence without any context. Then we follow up with essays on a given topic. From my experience it was mostly descriptions of physical things, experiences or phenomena. Describe in 150 words your favourite season. Write an essay about your latest trip to grandparents. Create a portrait of you mother.

And, oh boy, I sucked at it. I was seriously struggling to express myself and write anything original. In fact, for half of my school years my mother wrote my essays for me. She is a native Russian speaker, learned Romanian together with me while I was growing up. She would take the Russian - Romanian dictionary and try her best to compose a cohesive essay for my Romanian class. As a journalism graduate, she had a taste for books and writing. Of course, she shined when writing for my Russian class. My teacher never skipped an opportunity to listen to what my mother wrote. Presumably she thought I made up most of it and someone with more experience “helped” me.

And so it went. My mother liked writing in free form. She despised the template and was always trying to think of something captivating. When it came to foreign languages (I did English, French and German), my mother was trying her best, but word for word translations weren’t any good. So I had to come up with a way of composing something that was at least a bit readable (it is ridiculous my mum was doing my homework for me).

Throughout the years, I learned about the structure of a composition: the introduction, the body and the conclusion. My teacher of Romanian took it to the extremes saying that any sentence should follow the same logical structure. I learned about canned intros, closing thoughts, most used connection words. I was taught I can create a plan of what I want to cover in a composition. I was asked to dissect other people’s work and identify their works’ compositional parts. Obviously I learned from my mum, while writing down in silence as she was dictating word for word the essay to me. It all boiled down to a framework for writing.

Here’s how I did it. I would get a topic and research it, learn about any processes involved. Then make a plan, put down some bullet points. If it’s a description (and most of the times it was) pick up the dictionary and find enough adjectives and verbs that would aid that description. I would start with a few known words, identify good synonyms and repeat the process a few times until I end up with a word cloud that’s not too banal and not too obscure. Nowadays I use thesaurus.com.

Next I need to pick up some figures of speech. Good I had a list of these as well. Similarly I would choose a good intro template and adjust it slightly to not sound too boring. Follow my composition plan. And finally pick a good conclusion that fits the tone of the essay. Throughout the writing use the selected adjectives, verbs and figures of speech.

This is how I wrote my essay about the butterfly life cycle. I struggled being creative, and still do. I follow a set of rules that help me think things up. This is, of course, far from true creativity, but it works for me.

Who knows, maybe you’ll find this framework useful yourself. At the end of the day everything is a remix.

Django week_day Field lookup

2017-08-14T00:00:00+00:00

Django’s ORM offers built-in field lookups, one of which is week_day. So one can do Sale.objects.get(sale_date__week_day=2) to find out how well is the business doing on Mondays.

Wait what? How come 2 is a Monday? According to Python documentation, there are 2 possible weekday representations (identical methods are available for datetime objects):

date.weekday considers a weekday from 0 (Monday) to 6 (Sunday)
date.isoweekday() considers a weekday from 1 (Monday) to 7 (Sunday)

Whereas Django interprets the integer as 1 (Sunday) to 7 (Saturday). Here’s a short util to transform from Python to Django weekday value:

def to_django_weekday(date):
    return (date.isoweekday() % 7) + 1

Databases too have different interpretations for this value:

MySQL and Oracle are identical to Django’s weekday value - 1 (Sunday) to 7 (Saturday)
PostgreSQL - 0 (Sunday) to 6 (Saturday)
SQLite is identical to Python’s isoweekday() representation - 1 (Monday) to 7 (Sunday)

Even though the value varies between databases, it’s still not exactly clear why this value is so alien to any of the Python representations. I think I will pester some people today at DjangoCon US.

UPD: Thanks to Marc Tamlyn for confirming that there isn’t any special reason for the choice of weekday interpretation in Django (maybe MySQL). This code was last time updated in 2009 and, at this point, an update would be backwards incompatible.

NDB in-context cache behaviour

2017-07-19T00:00:00+00:00

NDB allows an App Engine app to connect to Cloud Datastore. It helpfully provides 2 types of caching enabled by default: memcache and in-context cache (also referred to as in-memory cache).

Timestamped

App Engine Python SDK 1.9.57

Anything wrong with it?

Yes and no. See this:

>>> box = LegoBox.query().get()
>>> box.pieces_count
32
>>> box.pieces_count = 64  # set the value, don't save
>>> LegoBox.query().get().pieces_count
64
>>> LegoBox.query().get(use_cache=False).pieces_count
32

When fetching a fresh object even within the same thread, one would expect to get the value from the database and not the one from an in-memory cache. This is considered working as intended.

A happy clueless developer could write this:

def test_smth():
    box = LegoBox(box_type='architecture', version='v1')
    box.put()
    box.count_pieces()

    # Refresh the box object and check the result
    box = box.key.get()
    assert box.pieces_count == 544

LegoBox.count_pieces has a bug in it where it forgets to call put (bugs happen y’know). Unless one is diligent to always disable the in-context cache for the fetches, this test isn’t good enough, because it doesn’t expose the bug. YOLO!

Sadly this behaviour is not mentioned in the official documentation, so there’s a high chance one will discover this “desired” effect in a bitter situation while debugging for hours.

How to disable in-context cache

First strategy is to pass use_cache=False when making queries.

Second option is to disable the cache per model. In theory this will slow down your app. In practice, I don’t have benchmarks, so I don’t know. Moreover it always depends on what you are doing.

class LegoBox(ndb.Model):
    _use_cache = False

Third stategy is to set a global policy to turn off automatic caching for all NDB models and be done with it.

# appengine_config.py
from google.appengine.ext import ndb

context = ndb.get_context()
context.set_cache_policy(lambda key: False)

I was a happy-go-lucky developer who wrote lots of passing tests that interact with NDB models, only later to discover that the code is failing in production. Luckily people smarter than me pointed out what might be the cause. This is a cautionary story to help one avoid such issues.

Humanizing among coders

2017-05-28T00:00:00+00:00

This is a transcript of the keynote I gave at PyCon CZ 2016. It’s a bit lengthy, but hopefully worth your attention. Alternatively there’s a video recording of the talk you can watch.

Hello! Ahoj! Salut!

There used to be a time when a tech-savvy, computer person was regarded as a reclusive night creature that lived in the basement and loathed going outside and talking to people. There are developers who prefer to work at night and don't feel very comfortable interacting with others. And that's alright. But the stereotype's been changing.

Similarly to how technology and some forms of programming have been creeping into the mainstream, the typical programmer has evolved into a social person, who likes going to conferences, meetups and hack days, give talks and generally enjoys the company of others.

We all know being a programmer isn't only about writing code, drawing diagrams and navigating documentation. It's also about planning, brainstorming, arguing design decisions with your team, giving and receiving feedback, joining client meetings and discussions, finding a middle-ground between clients' or communities' requirements and real world possibilities and being able to explain that.

Suppose our different sets of skills are a layered cake. Yum!

One big layer is dedicated to technical expertise...

... and another layer for social competence.

Truth is, we still prioritize technical expertise over social and emotional education. Just think about the plethora of books, tutorials and articles that teach us coding. Even the most niche little things, find a community of devs that care about it and want to share and learn.

Conversely the social aspect of what makes a good developer isn't referred to very often.

What does it actually mean to be a good developer from a social point of view?

Is it about smiling to each other all the time? Or is it about general courtesies like saying "Hi", "Bye" and "Thank you"? It's definitively about not insulting or harassing your team or community.

This talk isn't exactly going to be about that.

This talk is about the subtle "that wasn't quite right" moments and all sorts of invisible interactions that happen all the time between us and can inconspicuously shape the general mood around us. It's that slim layer of the cake.

This is our first story: "Beginners & Mentorship".

Meet Martina. Martina is the maintainer of open source project grepy. She handles the releases, changelogs and merging code into upstream.

This is Ionel and he is currently in his 2nd year at university, has been using C for a while and knows a bit of Python and JavaScript. He really likes grepy and has been using it for a while. He figured he's missing an itty-bitty feature - making grepy search for case-insensitive patterns.

He knew that the tool is "open source". He heard that open source is like getting access to some code to which he can make a modification and then share with everybody and everyone will be very grateful to him.

Ionel, all enthusiastic, firstly finds out what the IRC address of the grepy community is, joins and says: "Hi all! I'd like to contribute to your project. How do I get started?". Reply back: "Welcome :) To get started check the documentation on contributing in the repo."

Ionel cracks on with the code, spends about a week of free time in total to implement the feature and creates a pull request. In the comments he says "Hi. This is feature so and so and my first pull request ever. I'm a total beginner." Here comes the moment of truth!

Martina finally finds some spare time to review the code. She glances over the PR and leaves a couple of comments: "Please remove the loop and change this variable name. For the PR to be accepted we require unit tests and documentation. Back to you."

Ionel is crushed by this message. What the hell are unit tests? How do I add docs? What exactly should I write? In reality it took Ionel 3 days to figure out how to run the project and then another 3 to write those 6 lines of code. He can't understand why his code can't be merged as is. It works, right? Due to growing frustration, Ionel gives up and never makes the second PR.

In an alternate universe, this is what could have happened.

Martina glances over the PR and leaves a couple of comments: "Hi, thanks for your contribution. Good job on feature X. Please remove the loop and change the variable name. For the PR to be accepted we require unit tests and documentation. Back to you (smiley face)."

This is called sandwich feedback: say something nice, say something not so nice, and end with niceness again. It's somewhat better than the previous version, but it feels insincere, and people can feel that. We can do better.

In an alternate universe, take II.

"Hi, thanks for your contribution! We really appreciate having beginners on the project.

I see you are using a loop to make the string lowercase. That looks like a valid piece of Python code. You were probably not aware of that, Python has a built-in method for that called "lower". You can do 'aNyStRiNg'.lower() and it will convert the entire string to lowercase. Neat, right?

Don't worry about missing this, I still have trouble remembering all of those functions and methods from Python STL.

Thank you for adding this feature to the grepy. Usually when we write new features, we find it very useful to provide some short docs on them. If you look into the docs folder, there is a file called features. Can you please add a line mentioning the case-insensitive argument, that'll be very helpful.

Ohh, and we also usually write tests. There's a lot of stuff to know about tests. Start by reading this and that. Follow our guide on adding tests.

If you get stuck, just ping me on IRC and I'll try to help you!"

Huh, that was verbose. What changed? Everything!

Let's break it down.

The intro: "Hi, thanks for your contribution! We really appreciate having beginners on the project."

Be welcoming. As beginners, they have taken a very big leap to make that first step. Give them some encouragement to keep moving. And if you consider that your project is not well-suited for beginners, more on that a bit later.

Only say that you appreciate having beginners contributing, if you really mean it.

Seriously, be honest. If you're gonna say things you don't believe, don't even bother. People can sense when you're talking from a template.

If you're asking for a change, take the time to explain why something is important. Just stating facts sounds like giving orders.

And you won't believe how some things obvious things to you aren't obvious at all for beginners or even other developers.

"Don't worry about missing this, I still have trouble remembering all of those functions and methods from Python STL."

With this you've established a very humane connection. You've admitted of your own flaws, which makes you seem more real and humane, not a super-creature (if you've actually memorized all of the Python STL, then don't lie for the sake of the contributor to feel good about themselves). You've connected with something very deep within yourself to understand what the person on the other side of the computer screen experiences.

This ability to put yourself in other people's shoes, to feel with them has a name.

It's called empathy.

Empathy is about taking perspective, withholding judgment, recognizing emotion in other people and communicating it. We all need a little bit more empathy, both on the giving and receiving end.

Along with explaining why, also explain how. If it's a simple task, make it sound simple. Don't obscure by using fancy words or shoveling in some elaborate concepts from computer science. It's just one line of text in a markdown file. And offer help, beginners need our help.

That was a lot, wasn't it?

To be exact that's 168 more words than the first draft. But we know it's not about the quantity of sentences, right? It's about their quality.

To achieve this level of understanding and empathy, it takes a lot of energy. It takes enormous amounts of energy and emotional struggle for the beginner to make the PR and it's equally energy draining for the mentor to keep up. And when both sides can see each other's perspective, beautiful things happen.

I've set the focus on the mentor and their ability to be empathetic, because they have been beginners themselves and have a direct reference where to look, whereas beginners have rarely been mentors, hence lack that experience.

Don't forget to connect with your beginner self. Nobody is born knowing these things. We've all written bad smelly code and we kept on writing that bad smelly code, because nobody told us that we suck and we should never do that. Or maybe someone told you so, but you were stubborn, and brave, and continued regardless. But not everybody is. So don't discourage people. We grow professionally with time. Offer this time to beginners.

It's curious that almost every developer will tell you that when looking at their one year old code, they'll say "That is horrible, what was I thinking back then?". We accept our flaws, we know we are not perfect. Let's accept the fact that others aren't either and let them make mistakes.

Mentorship is not just about that. There's a whole lot more...

For me personally mentorship is an intersection of teaching, inspiring and guiding. It's hard work.

So open source...

I think beginners should have a safe playground to try things out without the pressure of perfectionism. Open source is often not the place, especially with a small, solitary, one maintainer, who probably also works full-time, projects. Lack of energy drives a lack of mentorship. Also consider most of us are not getting paid for it and even being shouted at by random people on the internet for not fixing a bug in time, because, you know, they need it.

What fascinates me is that there are projects that harvest the energy for mentorship. Often those will be big community, many contributors type of projects. These are the kind of projects that can afford to invest in good beginner documentation, that have people who can do so, who will often participate in Google CodeIn, Summer of Code camps and such. I want to say thank you :)

If you are a big community, but you prefer to neglect beginners, then you are losing some unique perspectives to improve your project and to self-evaluate.

Let's talk about legacy, also nicknamed "code archeology".

This is Lela. Lela has been working on this legacy project for the past 2 months. The more she got into it, the more she realized how big it is. It's been developed in the company for the past 3 years and before that it belonged to another agency, having in total about 50 contributors.

Lela has a bug to fix and first thing she does she writes a failing test and then git bisects until she finds the hurting commit.

Aaaaand the commit's author is herself. It looks like this is another fix that she did when she joined the project.

Going back in time when she first tried to fix whatever she was fixing, we can see how she acted. It was a side fix, she saw something that looked like an awkward way of accomplishing what she thought it was supposed to accomplish.

So she git blamed. Saw it was a person who already left the company, made a snarky comment to herself: "Ha, is that proper Python? Must be the reason someone got fired." So she changed a couple of lines of code as a matter of fact. As we know, it backfired.

In an alternate universe, what could have happened?

Lela finds the odd looking piece of code, does git blame -s /path/to/file.py, because at this point it actually doesn't really matter who made the change. Finds the commit. Looks at the commit message and commit contents. She is gaining more context.

Seems like this bit of code was moved from somewhere else, so she does git blame before that commit - goes deeper into the history.

Finds out a very nice explanation on why it looks the way it looks and what it's supposed to serve. She found the root!

Lila leaves a comment in the code, summarizing everything she learned and why this shouldn't be removed. Case closed.

Git blame is an excellent tool if used for things beyond just blaming people.

Some people have strong feelings about the choice of words - blame.

We have an alternative. We can create an alias to git blame. Quoting from the original repo of this git praise alias: "Sometimes you find the most amazing piece of code and want to know who to thank for this wonderful contribution and then it feels wrong to use git blame."

But what if you personally rarely use git blame to literally blame someone or git praise to thank people (though that's definitely nice)?

Hence I came up with this alias - git travel. Because for me it's all about going back in time and like an archaeologist figure out why things are the way they are. It's not about me and my judgment, it's about the history. Most probably nobody is playing dump, there is a reason for this bit to exist.

Context makes a very big difference and that's why it's worth spending time to understand project history. All code is legacy or will be some sort of legacy soon enough (you don't need decades for that), otherwise we wouldn't have big and complicated systems. So knowing your way around legacy is a skill we should develop.

Couple of years ago, while at university, I stumbled upon this article that discussed a very complex codebase of a system. And it's presented from the perspective of the new architect that decides to makes a big revamp of some core components. Why? Because they wanted to make it better.

After 3 years into the project, they realized that was not the way to go. And the main issue that was identified is the fact they didn't try to understand in-depth the reasons behind the initial design solutions to understand the basic problems that it tried to solve. We often see only the tip of the iceberg.

That doesn't mean we shouldn't improve our systems - refactor and improve our code! It just means we better be aware of the history before trying to write the future.

Interestingly enough while traveling into a project's past, you realise what a commit message should be and what it shouldn't be: it should explain why, not what (because I can read the code, I can see that the variable got renamed, I want to know why). And that suddenly makes you realise how terrible you are at writing commit message. You can correct yourself.

It's incredibly useful to tie commits to a ticket. That opens a new dimension to understanding the context.

Finally as a last resort, if no meaningful artifacts are found, check the author and if they are still around ping them and ask why.

This is my code travel guide. I hope you'll find it useful.

Having talked all about git blame, made me think of another story.

Imagine a regular day at the office. Suddenly the reporting tool starts flashing red, everybody is getting notifications on every email, the alarm in the room goes off, there are lots of 500s happening on production. Panic-panic! Team gets together really fast, figures out what the problem is and hotfixes the issue on production. Everything is back to normal.

Now stage 2 begins: who's to blame and who should we punish?

The team figured it was Miloš who made the commit that broke everything. (At this point the room burst into laughter and I am completely clueless why this is funny. Apparently this is an inside joke - Miloš is the first name of Czech Republic president and he is not a very popular figure.) Everybody starts pointing fingers. Miloš is not fired or removed from the project, but the management is giving him the looks. It's cringy. Ivona, his teammate, even said: "I always thought Miloš is sloppy and something like that will happen eventually." Everybody settled with the root cause of the problem - Miloš, and now they can move on.

This tendency for people to place an undue emphasis on internal characteristics of Miloš (as being sloppy or inattentive), rather than the external factors, in explaining the situation is called the attribution effect. We want to find a simple answer to a complex problem.

Did we actually solve the problem? No, we didn't. Because anyone can be Miloš.

Therefore in an alternate universe.

After fixing the bug, the team gets together and has a different post-mortem. They ask questions: "Why did it happen in the first place? How did it slip into production? How can we prevent these kind of problems in the future?". And they try to find solutions: "Maybe we need better tests, or CI, or a different way of doing code reviews."

Don't find fault, find remedy.

This could happen to anyone. We are humans, we make mistakes. That's why we make up rules and build tools to prevent us from making those mistakes.

That said, we should not keep people that are dragging the project down. Sometimes we have to say no to some people. How we decide who and why should be let go is a complicated matter and I believe every company has its own process for that.

Our next story is about passive aggressive interactions.

This is Özlem. She joined a new team. How exciting!

And this is Mărioara. She's been in the team for the past 2 years - proper old-time team player.

Özlem is excited to get started. She picks up her first ticket, investigates, writes the code, writes the tests - all good. Puts it in code complete.

Mărioara picks it up for a review. She changes some implementation details splitting some functions and rewriting a method. She then proceeds to merge that code and set it as done.

Özlem looks at the changes made, feels a bit awkward to ask questions, so she lets it go. She thinks to herself: "I should probably be more attentive next time."

In a few weeks time Mărioara again picks up one of the Özlem's tickets for review and again moves things around, rewrites variable names, etc. Again no real comments from her. That happens again and again. Özlem remains silent to her growing frustration.

Now let's see the perspectives of both parties involved.

Özlem feels like she isn't even contributing to the project. She can't take ownership of anything on the project, because some bits of her work are being rewritten all the time. She never gets any real feedback on what was wrong and why this different solution is better. She didn't want to speak up, since she was new to the team. Didn't want to be the trouble-maker.

Mărioara does a code review and sees some parts that could be improved, not that the original solution was wrong. She has this itch of just making things perfect. She think it's just faster (hence more beneficial to the project) to fix those things herself, rather than leaving long comments and explaining her vision. She never considered that the author of the ticket is offended or feels alienated.

In an alternate universe.

Mărioara reviews the ticket and sees a main point that could be improved. She goes to Özlem and says: "Hey, I was reviewing this ticket and I think it's a bad idea to store that value in the database, since once it expires, it's no use to us. Do you think we might change that and store that value as a class attribute?"

"That's a good point. But the problem here is that we instantiate that class in two other places. So we will have to fetch that value twice, which is expensive.", says Özlem.

"Right."

Özlem goes again: "We could store it in memcache."

"Good idea!", says Mărioara. "Wanna grab the ticket again or do you want me to do it?"

"I'm free now, so I'll gladly amend the feature."

"Nice. Ping me after you're done if you need a review."

Özlem and Mărioara manage to truly collaborate and come up with a better solution than either of them had in mind. Also both feel included. It's so easy to say: "Hey, I have this concern." It's also about offering a choice.

At this point it doesn't matter who will finish the ticket, because they have an agreement and both are looking into the same direction. That way spirit of collaboration and trust can flourish in a team.

Here we are at our final story - the power of we. We as in we together, not the royalty.

This is Haru and Vlad. They work together.

Haru and Vlad are estimating a new feature. On a specific line in the estimate Haru says: "Max wrote this parser a while ago. I wouldn't use it for this feature, because it's meant to handle only very small sets of data. Plus I constantly find bugs in it. You could rewrite it Vlad."

"Alright, I'll make sure I also use the new conventions that Amira introduced recently."

In an alternate universe. You probably see where I'm going with this...

I made an itty-bitty correction. Replace all the pronouns with "we".

"We wrote this parser a while ago. We shouldn't use it for this feature, because it's meant to handle only very small sets of data. Plus we constantly find bugs in it. We could rewrite it now. What do you think Vlad?"

"Sound good, we need to make sure we use the new conventions that we've introduced recently."

"We" gives a strong feeling of togetherness.

This is such a trivial change. But the tone changes straight away. It's a big shift in our heads of how we think about our work and what a team is. We are together in failure and success.

The I vs he/she/they idea extends to communities build around tools, libraries, programming languages. I - part of Python community versus them - from Java/PHP/Go/whatever community. I'd be very glad if we could change the versus to together. And let's not tolerate belittling of others.

I've made all of those mistakes and more. But I believe it's possible to change and I'd like to encourage people around me to change for the better as well.

Influence and changes to the norm of interacting in a team or open source community can be nourished in different ways. One possibility is to have it imposed by a general rule coming from higher up management. But I find that true long-lasting shatterproof rules of interactions come unwritten kicked off by individuals themselves in their everyday social encounters with other people.

Things tend to accumulate and little by little good things and healthy interactions will overflow into something good - like a pleasurable place to work.

All these little stories are just a fraction of what we experience every day on a social level. These stories won't give answers to many situations that we find ourselves in.

That's why you should let empathy and compassion be your guide.

Not only in your professional life, but also when you're driving, getting tomatoes at your local corner shop, visiting family for holidays or ordering pizza.

Thanks! Děkuji! Mulțumesc!

Code review checklist

2017-02-21T00:00:00+00:00

Writing code is hard, reviewing it, is equally hard. In my team every ticket is scanned by a second pair of eyes.

Why? Half-assed work can make a company look bad, lose money, lose clients, generate stress and extra work, thus a good code review is a must before going live.

Over the last couple of months, I’ve developed my own internal code review checklist. I use it both for reviewing for own finished code and my teammates code complete tickets. It’s split up into 3 sections: code, automated testing and manual testing.

Code

Does it compile? Look for obvious errors, most IDEs will spot them for you. Baffling that these kind of issues slip into code completes, but we are all humans and mistakes happen. Most common reasons: last minute untested change that surely won’t break anything or forgot to push the fix.
Can you spot any runtime errors just by looking at code? Try to reproduce the exception.
Is the feature/bugfix complete? Does it tick off all the requirements? Is there anything missing or misbehaving?
Is this a good solution? Is this the best solution out of all those that have been considered? This is mostly related to system design and general approach to problem solving. The definition of good and best is context dependent.
Does it introduce any security holes? Definitely not something to be disregarded.
Is the code efficient in whatever metrics matter for your case?
Is it logically correct? Does it solve the problem? This is usually central to the code review, hence obvious, but I don’t want to skip obvious steps.
How does it affect the rest of the system? We review code diffs with no or minimum context regarding the rest of the codebase. With diffs you can only see that much, thus how about start looking in other places? Think about all the other components that might depend on this change. Where any of the APIs violated?
Is it within the scope? Does the ticket touch any unrelated parts of the codebase or does side fixes? The more it gets cluttered, the more complicated it becomes to review, test and release. Stay on topic.
Are there any code style issues? Conventions are important for the sake of codebase sanity. I know we all have strong opinions about all the things, but for your team and code well-being, please disagree and commit.
Is the code readable? Can someone else understand it? How long did it take you to figure it out? Do all namings make sense?
Does it have good and enough documentation? Not to be ignored or considered of least importance.

Automated testing

Does it need tests? Sometimes a favicon update will do just fine without tests.
Does it have tests? Seriously, write tests.
Are tests logically correct? Tests can have bugs too. Review tests thoroughly.
Are tests simple, easy to read and debug? Forget about DRY, don’t build a whole infrastructure to test your code (unless you are writing an isolated testing framework with a clear API that everyone else on the team is more or less familiar with), KISS (keep it simple, stupid). If something fails, it should be apparent what broke.
Are tests testing the right thing? Tests are not a checkbox to be ticked off, they are your safety net. Just the existence of tests or higher test coverage doesn’t guarantee a bug-free life. Make sure to get a good insurance with a solid testbase.
Are tests testing the edge cases? Best case scenarios suck!

Manual testing

Can you run the code? Don’t skip that. Automated tests look good on paper (ehhhm, computer screen?), but running the project is still a good idea. Automated tests reduce manual testing to the minimum, but don’t exclude it completely.
Can you follow through several test scenarios? Pick obvious and non-obvious ones: do they succeed?
Is the rest of the system looking/working fine? Links to How does it affect the rest of the system? from Code section. Similar, but more straight-forward. Have a click-around and make sure things are fine.
Can you break it? Try to think of ways it might fail: feed it large sets of data, consider third-party systems downtime and how it can affect you, use an unusual email address (did you know email address can have comments in-between tokens?). Don’t worry, your users will still find ways to break it for you.

Feel free to suggest more checkpoints :)

Keynote animations: express yourself

2016-12-15T00:00:00+00:00

Keynote might not be the best tool to create animations, but it’s good enough and provides most common animation tools. Moreover you don’t really want to open After Effects and export a movie just to make an object wiggle a bit during your presentation. It’s also an interesting challenge to create something complex out of something trivial. Constraints foster creativity.

Meaning

The main goal of an animation is to convey meaning. Think of it as another dimension to express an idea. Try to remove the animation and see if you can communicate the same message equally well. If you loose clarity, then this is a meaningful animation.

In this case, the animation is used to convey playfulness. “Potato is a serious company, but we also like to be silly.”

Don’t overdo it

In conjunction with the previous statement, don’t fill up your presentation with useless animations. When things move, spin, burn too often, it’s hard to follow along and justify any of this live action on the screen.

On a more practical note, certain types of animations might even make some people dizzy - I’m looking at you Prezi.

Oh you, flame effect, I will never use you, unless I’m going for on-purpose tacky style.

Baby steps

So how to make the first slide with the smiling Potato? First solution: use 2 slides, one with a plain Potato and another with a Potato with a big smile on it. Better solution: apply an animation called “Appear”, found in the “Build In” tab of the Animate section. Click/tap and, voila, the image appears!

Bullet points

They are haunting us in every presentation! They are useful though, as they reduce long complex sentences to a few important key points. And if you want to make them appear in a special way, say one by one, Keynote has got your back.

Don’t even think making 5 separate slides and placing on each plus-one bullet point. This easily gets out of sync when you decide to add or remove a bullet point and now you have to edit not one slide, but 5 (worst-case scenario).

These is an exception to the rule though. If the speaker notes for the first 2 points are already too big and in presentation mode you need to scroll to get to the last sentence, then it’s a good idea to split up the notes into a couple of slides and hence duplicate the content. I should start doing that ʕ⊙ᴥ⊙ʔ

Center of attention: highlighting

As you speak you might want to direct the attention of your audience to a specific line of code, element or word. This is a neat trick, because it allows the viewers to grasp the general picture at first and then dive deep into certain segments. There are many ways to do that. In case of code I like to use highlighting.

Create a shape and give it enough opacity to make the text underneath eligible. Then apply an appropriate “Build In” animation to the shape - I like “Move In” with a little bit of bounce. It’s hard to ignore such an animation, and at the same time it doesn’t make me cringe (like “Bouncy”).

Tip: assess animation duration. The animation should be fast and noticeable. In most cases 1s is too long. Experiment with 0.5 - 1s range and see what feels best to your presentation pace.

Center of attention: opacity

Similarly to highlighting, you can use opacity to make something stand out in the midst of blurred/semi-transparent content. I remember seeing the trick for the first time at DjangoCon Europe 2015 in Cardiff in Ola’s talk Pushing the pony’s boundaries and I really liked it.

“Opacity” animation can be found under the “Action” tab. Since I prefer to almost omit the transition of going transparent, I set the duration to 0.1s.

An important take-away from this short video is the “Build Order” dialog. This is the control panel of all animations, where they can be deleted, grouped and rearranged.

By the way, alternatively to copy-pasting animations, it’s also possible to group elements together and apply a single animation. Unfortunately for the current 3 elements when grouped, the animation doesn’t apply well. Would be fun to aggregate and post all Keynote bugs I’ve discovered while preparing for my talks.

Multi-actions

Things get complicated when you want something to go opaque, then transparent, then opaque again. In other words, apply multi-actions to the same element. It’s somewhat simple - click “Add Action”, select another animation and configure it. My problem with that is the UI that is showing only the last applied action. To be able to go to older layers of animation, you need to open “Build Order” and operate from there (that was so not obvious at the beginning).

A process

A single animation is worth a thousand words when modeling a process. It’s like when they spend 5 pages to explain how an engine works, and you still don’t get it. But once you see the animation, it suddenly becomes clear.

Here I explain visually how tests are run in parallel, simplified version taken from my Testing in Django talk at Django Under The Hood 2016. This is a showcase of what is possible and how it can be achieved using Keynote animation toolkit.

Now you are ready to unleash your creative freedom and deliver jaw-dropping presentations. No flames please.

Intricacies and optimization of Datastore indexes

2016-11-28T00:00:00+00:00

It’s nice that development server is so kind to generate Datastore indexes for us, unfortunately those aren’t always optimal and might require some tweaking from our side. Hence it’s important to be knowledgeable on intricacies of index creation.

This is a follow up post to Datastore indexes basics that covers essentials, mentions the dependencies used and contains some base models that we will be using here.

Speed considerations

The reason why we want to optimize indexes is speed. Simply put the more simple and composite indexes an entity has, the slower it becomes to do writes. That’s why Datastore imposes limits on the number and size of index entries (200 composite indexes and 2 Mib maximum sum of the size of an entity’s composite index entries). It is possible by having a large number of properties on an entity/model or having a huge amount of composite indexes in index.yaml to exceed those limits though.

When order matters

Order in the WHERE clause doesn’t matter. If you have multiple equality filters on different properties, then those 2 indexes are equivalent and one of them can be safely discarded. Going back to our example with a Book model. If we execute this query:

Book.objects.filter(author='Mark Twain', title='Some title').order_by('-published_date')

Development server will create this index:

- kind: library_book
  properties:
  - name: author
  - name: title
  - name: published_date
    direction: desc

It can serve this query as well (we have swapped in place author and title properties):

Book.objects.filter(title='Some title', author='Mark Twain').order_by('-published_date')

It is possible to confuse automatic index creation and make it also create this index:

- kind: library_book
  properties:
  - name: title
  - name: author
  - name: published_date
    direction: desc

Then it’s our duty to spot indexes alike and remove duplicates.

In general Datastore will try to create and insert indexes in alphabetical order with properties inside of an index ordered like that:

Properties used in equality filters
Property used in an inequality filter (there can be only one)
Properties used in sort order

That said, the direction of a property makes a difference, hence order_by('published_date') is not the same as order_by('-published_date') and 2 indexes should be created respectively.

Minimum vs perfect

The index that we created above is a perfect index - the whole query is served by a single index. This allows the most efficient and fast querying. In contrast we could split the perfect index into 2 minimum indexes:

- kind: library_book
  properties:
  - name: author
  - name: published_date
    direction: desc

- kind: library_book
  properties:
  - name: title
  - name: published_date
    direction: desc

These 2 indexes now serve 3 different complex queries:

Book.objects.filter(author='Mark Twain').order_by('-published_date')
Book.objects.filter(title='Some title').order_by('-published_date')
Book.objects.filter(author='Mark Twain', title='Some title').order_by('-published_date')

To serve the last query, Datastore uses zigzag merge join algorithm. To be able to merge those 2 indexes to obtain results for the last query, both indexes have to be ordered the same way.

It might not seem like a major winning strategy. In our case we have reduced 3 indexes to 2 - not a big deal. But consider a different example where we add more properties to the Book model and we create an interface where librarians can choose by which properties to filter and to sort. If we try to predict every single combination and create perfect indexes for each, we will obtain fast querying, nevertheless we might easily reach the index cap and the operation of creating or updating books will become slower.

Consider we have 6 properties for filtering and we allow ordering by 4 properties. That’ll result in 256 perfect indexes (whoops, we’ve reached the index cap). It’s computed the following way:

2^(number of filters) * (number of different orders)
2^6 * 4 = 256

In this case we should resort to creating minimum indexes or a combination of minimum and perfect indexes. The absolute minimum to serve all the possible queries in this case is 24 indexes (6 * 4). This might not be the optimal solution though, some queries might be executed more often than others, so maybe it makes sense to add another 3-4 perfect indexes to serve most common use cases.

Besides when resorting to minimum indexes, we should be aware of the shape of the data. Merging using the zigzag merge join algorithm comes with a cost. Poor performance is achieved when many entities match each scan in separation, but few entities match the query as a whole. App Engine has an excellent explanation on that in Datastore Index Selection & Advanced Search article.

Exploding indexes

Iterable fields complicate things. Normally each row in the index table represents one entity in Datastore. Take this index Index(Book, author, title, -published_date). If we have 10 books with those properties set, then Datastore will create 10 rows in the index table.

If we add another indexable property to Book keywords = ListField(CharField), then the number of rows becomes the number of unique values in the ListProperty. Datastore will create an entry for each combination of property values.

Now if we use multiple multi-valued properties, then the number of entries for each entity can “explode” combinatorially. Even though it’s a single index, such an index can dramatically increase the cost of writing an entity to Datastore.

Let’s amend our Book model:

class Book(models.Model):
    title = models.CharField(max_length=200)
    ...
    keywords = SetField(CharField)
    ratings = ListField(DecimalField)

And create another index:

- kind: library_book
  properties:
  - name: title
  - name: keywords
  - name: ratings

With the following new instance:

book = Book.objects.create(
    title="My sci-fi book",
    keywords=['sci-fi', 'dystopia', 'post-modern'],
    ratings=[4.3, 4.9, 3.9, 4.1, 4.6]
)

To ameliorate such a hungry index, we can split it up into 2 indexes (same zigzag merge join algorithm applied in case of a query that filters and sorts on those 3 properties):

- kind: library_book
  properties:
  - name: title
  - name: keywords

- kind: library_book
  properties:
  - name: title
  - name: ratings

Unexploding indexes might not always be a necessity, since (again) it all depends on the shape of your data. You might be fine with a single index that contains 2 iterable properties if one of those is a property that always has only one value. Maybe it was created with a thought about distant future, where more options can be added without the need to migrate all the data.

Dead indexes

Large projects with a lot of models might suffer from dead-index weight. Adding indexes is easy, removing them - not so much. You never know if there’s a hidden query somewhere that your search through the codebase didn’t spot. So maybe we should just keep that index? Nope.

There are different scenarios when we can and should identify dead indexes:

Removing a query. That query might have required an index in the past. We can delete it, but we also need to check the rest of the code if nobody else is using this index. It can be also be a minimum index used in a mix with other indexes to serve a complex query.
Changing Django Meta class ordering. This kind of change will have repercussions all over the code and most probably all the indexes will have to be updated. Big changes ahead!
Scanning for duplicate indexes, whether it’s obvious duplicates with properties in different orders or hidden duplicates with equivalent indexes just in a different form (think minimum vs perfect).

Either way we need to practice mindfulness to keep our indexes optimal and at a minimum.

Datastore indexes basics

2016-11-20T00:00:00+00:00

If you’ve been developing apps to host on Google App Engine (GAE) and use Google Cloud Datastore database (further referred to simply as Datastore), then you are probably aware of indexes. Simply put, indexes are what makes reads faster with the caveat of slower writes.

Timestamped

This is a section that I want to include more often to ground any of my explanations into a specific time-dependencies frame.

App Engine Python SDK 1.9.40 - 2016-07-15 (examples are in Python, though the concepts are language agnostic and applicable to other GAE environments)
Python 2.7 (because App Engine Python SDK doesn’t support Python3, the issue about that has been open since 2008)
Djangae v0.9.6 because it’s the easiest way to run Django on GAE and I’ll be using Django’s ORM integration with Datastore instead of ndb (still same concepts apply regardless of who makes the query)

Simple indexes

Say we have the following Django model:

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.CharField(max_length=200)
    published_date = models.DateTimeField()

App Engine will define a single index on each property of an entity (model in this case), except for TextProperty and BlobProperty. Each index has a direction, by default ascending. In our case it means entity Book will have 4 simple ascending indexes (3 explicit properties and implicit pk).

What can we do with only 4 indexes? Well, lots of things: fetch all the books, query books using only equality filters or only inequality filters, sort only by a single property with no additional filters. App Engine docs contain the full list of allowed queries for automatically predefined indexes.

Composite indexes

What if we want make a more sophisticated query:

Book.objects.filter(author='Mark Twain').order_by('-published_date')

In this case Datastore can’t return a result based on the simple indexes. It requires a composite index. All composite indexes go in an index configuration file named index.yaml. When uploaded to production server, App Engine will parse this file and create all the indexes based on this configuration. Already existent indexes will not be recreated.

Generating composite indexes

We wrote a complex query. So far App Engine doesn’t know about it. For the system to be able to recognize a new composite index, we should execute the query on the development server. This will trigger an automatic write to index.yaml of the perfect index in the section bellow this line # AUTOGENERATED.

To be noted: indexes will be not created while running Django tests, because Django test framework will not run the server during tests (it interacts directly with the WSGI interface to produce requests and responses, so dev appserver is not involved).

Furthermore indexes can be manipulated manually, by listing them above # AUTOGENERATED line in index.yaml. We have the power to decide what’s the best index to serve our query.

Require indexes

If we want to be aware of what indexes are getting created, rather then letting the dev appserver do its automatic index creation, we can configure our dev environment to throw NeedIndexError exceptions when it’s missing an index. If running the server using the App Engine command, then we can do:

dev_appserver.py --require_indexes

If using Django’s/Djangae’s management command to run the server, then there’s no easy to do that.

./manage.py runserver --require_indexes  # won't work :(

That won’t work because Djangae isn’t passing command line args to the dev appserver args parser and there’s an open issue about it (if by the time you are reading this, this issue is closed and has a merged patch, then you might be lucky and that command works).

Hacky workaround to just make it work: go to google/appengine/tools/devappserver2/devappserver2.py and replace datastore_require_indexes=options.require_indexes with datastore_require_indexes=True inside DevelopmentServer._create_api_server. Sorry, I just grepped (actually aged) inside Python SDK and picked up the first convenient place to force this setting. Maybe there’s a better way ¯\_(ツ)_/¯

Therefore now if we want to run this query on dev server, Book.objects.filter(author='Mark Twain').order_by('-published_date'), we will get a pretty error saying:

If this index fits us well, we can copy it to index.yaml. Most important we are aware what index got created and what query triggered it.

Cleaning indexes

Consider that we have pushed this index file to your production server.

indexes:

- kind: library_book
  properties:
  - name: author
  - name: published_date
    direction: desc

After a month or so we get a notice that librarians are not using this feature, so they ask us to remove it. Instead they want to be able to filter by author and sort by book titles in alphabetical order. This request yields another index. Since the old query has been removed, we remove the old index as well and hence end up with this index conf:

indexes:

- kind: library_book
  properties:
  - name: author
  - name: title

Once pushed to production, App Engine will build the new index, but will also keep the old indexes, ignoring the fact that we’ve updated the configuration file. This is intentional, since other versions of your app might still be using this index, so we have to be explicit.

Once we’re sure we don’t need any of the removed indexes, let’s run appcfg.py vacuum_indexes. We should be careful what we vacuum clean, since a missing index will result in a 500 for our users.

There is a lot more to say about indexes and various intricacies related to them. Now that you know the basics, we can delve into more details in my next post Intricacies and optimization of Datastore indexes.

A present for you, and you, and you

2016-07-23T00:00:00+00:00

Not here to try to sell you best Darth Vader mugs, perfect gift for your loved ones - only $9.99! Or am I? Haha! NO. I’ll be sharing how I’m responsibly prepping for my family and friends special occasions (think mostly birthdays).

Yes, often I’ll resort to a typical birthday gift, that doesn’t require any of my time or brainpower. I console myself there’s nothing bad in gifting a birthday card, a nice body cream, a funny T-shirt, a kick scooter, a kite, right?

But there’s this whole other level of coming up with soulful presents that require a lot more effort than just clicking “Buy now” on Amazon (don’t forget to enable 1-click Ordering to skyrocket your purchasing power or even get an ~~Echo~~ Alexa). So here’s the list of presents that I’ve given to other people and wish I could receive myself.

Fly high

Rough estimate of consumed time: 3 hours
Resulting effect: nobody can read my boyfriend, so it’s basically hard to know
Gift specimen: experience - safe and exciting enough

This one is actually a developed-world present, but it has a nice touch to it. Since we have this whole trend of giving people experiences, rather than objects, plus my boyfriend is very minimalist with his possessions and he already has everything he needs, I decided to get him a Trial Flying Lesson in a two seater aircraft. Once in the air the pilot lets you take over the yoke and pedals and fly the plane yourself. I knew my SO never flew a plane and wanted to.

I bought the voucher online. It was too late to order the special birthday pack that comes by post, so I decided to just print the page and make a cute origami shirt + mandatory birthday card with cats, which has handwritten inside “Open the shirt”.

Jojo in the stars

Rough estimate of consumed time: 3 days
Resulting effect: ecstatic rainbows spurring out of the mouth
Gift specimen: dangerous, thrilling and very teenagy

Ok. When I was a teenager, it was “cool” to be all bummed out about everything. Don’t judge me. And so I had this friend who liked this dark short animation video Jojo in the stars (I tried rewatching it now, but meh).

Jojo has a sad fate. I decided to light up the mood by making a graffiti of Jojo (he’s geometrically shaped and easy to draw) and give him wings, so that he can fly and save himself!

The logistics were simple: one white and a couple of black spay paint cans. Grab my friend, go under the bridge near the train tracks and start being all artistic (never actually did graffiti before). Now in retrospect I wouldn’t recommend to anyone going under the bridge when you’re a 17 year old teenage girl without event mentioning it to parents or any close friends. Nevertheless it all ended well. And my friend was ecstatic to see his birthday gift.

I’ve spend one hour trying to find a picture of the graffiti: nothing on my old photosharing account, nothing on google, nothing on any of my old blogs T_T

Doodled mug

Rough estimate of consumed time: 3 hours
Resulting effect: I think she liked it
Gift specimen: boring overall, but with a nice very personal touch

Mugs are a boring gift (I’ve just mentioned the Darth Vader mug at the beginning of this post). But… but this one is cute, because I covered it all in silly childish doodles. Used my acrylic paint. For a long lasting effect you can cover it in a layer of glaze.

Blossom in winter

Rough estimate of consumed time: half a day
Resulting effect: getting someone out of depression is a pretty good outcome
Gift specimen: romantic and almost environmentally friendly

It was one of those sad, cold, snowless winters. My friend used to live not far from the forest slash park. The bare trees in this time of the year look very gloomy and unwelcoming. I dragged my friend into the forest (no, I’m not a rapist), even though he kept on saying that’s going to make things worse and I kept on saying it’s going to make things better.

It made things better because I decorated a tree by the lake with pink flower-shaped post it notes. Few hours before meeting my friend I hooked the tiny pieces of paper by strings to the branches. Also scattered a few “flowers” forming a path to the tree. It looked like a little island of spring (⌒▽⌒)☆

A wave of good emotions. So good I forgot to take any pictures. Also no smartphones era.

I give you my hand

Rough estimate of consumed time: half a day
Resulting effect: someone was very impressed with my doodling skills (not sure if that was actually offensive)
Gift specimen: cute, useless and doesn’t take up much space

Connecting with my creative self again. On a A4 ~200gsm sheet of paper with antique finish I outlined the shape of my hand with a pencil. Cut the hand form. One side of it is feminine: soft shapes flowing in chaos. The other side is boyish: funky, hip with video games and robots. Nothing too special, but the execution was good and I know my friend kept it for a really long time (might still have it, but we don’t keep in touch).

Post-it everywhere

Rough estimate of consumed time: 2 hours
Resulting effect: big smiles
Gift specimen: all about how much I care about you mom

When I was a kid and didn’t have money to buy mom flowers, perfume or fancy creams, I’d make cards (handmade crazy looking “Happy Birthday Mom” cards). One year I wanted to make something different, so I used my pack of post-it notes and stuck them all over the flat with various inscriptions on them, things like “You cook the most delicious meals”, “Thank you for taking care of us”, “You look beautiful every day”. My mom mentioned a few spelling mistakes I made, but I’m sure she loved it (≧◡≦)

DIY robot

Rough estimate of consumed time: 1 hour
Resulting effect: he didn’t build the robot, but is still excited about the potential thought of it
Gift specimen: guided geekiness

This is the time when first Raspberry Pi came out. So I got one for my friend. In addition to it I wanted to give him all the tools to build his first robot. I found pibot kit that has all the elements and also you a step-by-step guide how to build a robot. They changed their website ever since and also posted better instructions.

My friend fiddled with the Raspberry Pi and other components, but never build the robot. At least I tried…

Profiling a Django App hosted on Google App Engine

2016-03-19T00:00:00+00:00

Ever used Django with App Engine? How about the Datastore? Well, if you didn’t Potato actually developed an in-house solution called Djangae that makes Django play nicely and almost seamlessly with the Datastore via its ORM. As if you are writing an app that’ll work with a SQL database, except you don’t.

If you go down that road, how do you figure out why a piece of Django code is slow? This brings us to Appstats.

What is Appstats

Appstats is a profiling library that comes together with the App Engine SDK. Its only purpose is profiling RPC performance of apps. The App Engine Remote Procedure Calls include:

Datastore calls (Datastore.Get, Datastore.Put, etc)
Memcache calls (memcache.get, memcache.set, etc)
URL Fetches
Mail sends

Why do you need it

Django offers a lovely ORM to abstract our code from a specific database. It has tremendous benefits and (at least) one big drawback. It hides the complexity of database queries. It’s easier to make a very expensive query (or make it even multiple times during a request) without realizing it. That’s why profilers exists: find out why an app is slow.

There is a popular library called django-debug-toolbar. It shows developers the raw SQL queries and the time it took to perform them. Unfortunately it’s rather useless when it comes to profiling an App Engine app - remember there is no SQL. But thanks to App Engine developers, we have Appstats.

How to use it

It’s actually incredibly simple. To enable Appstats you need to update only 2 files:

# settings.py
MIDDLEWARE_CLASSES += (
  'google.appengine.ext.appstats.recording.AppStatsDjangoMiddleware',
)

# app.yaml
builtins:
- appstats: on

This is optional, but highly recommended:

# appengine_config.py
appstats_MAX_STACK = 50

Since Django is a multi-layered framework, some of the calls recorded by Appstats can have very deep stacks, so the default value of 10 won’t be enough to reach our own code. You’ll see for yourself soon.

Appstats console

Once the middleware and the builtin are enabled, you can view the Appstats console at http://127:0.0.1:8000/_ah/stats/. If you bother, this URL can be changed.

Welcome to the ’90s!

As you can see there are 3 components of the console:

RPC Stats - it will tell you how many of each RPC calls are made overall.
Path Stats - how often you’ve hit a specific URL.
Requests History - all the requests that were made once you’ve started your server, the most recent being on the top.

Let’s focus on the Requests History - this is where the fun part is. From this list you can already see plenty of interesting information. Every request has a short summary: real=410ms api=0ms overhead=2ms (17 RPCs, cost=0, billed_ops=[]). It shows the total request time 410ms, the overhead 2ms of recording this request (Appstats does come with a little cost) and total number of RPC calls that were made during this request - 17 RPCs.

Click on either of the requests from Request History list and you’ll see this:

This particular example is boring, because I’m making a request to Django admin. But still this looks very helpful. You can instantly see the most expensive RPCs here. If you expand an item by clicking on little + button or directly on the rectangle from the timeline, it will show you the full stack. There are more than 10 lines there, right? Enough to actually go beyond the googleappengine and django code, and get to your app’s codebase (not in this case, since no custom code is involved) - because that’s where most probably stuff is slow. Thank you appstats_MAX_STACK!

Once you are there, inspect the code. Maybe you are making a terribly expensive query or you are writing too often to memcache (without a good reason), or you are not fetching from memcache when you should be, or you are creating a hell lot of deferred tasks, or… there can be lots of reasons!

Can I use it in production?

You could, if you want to identify a problem that is production specific, but no real reason to leave it enabled, because Appstats will only store the last 1000 requests. Moreover they have that tiny overhead. In conclusion, use Appstats extensively during development and only occasionally on production.

Happy profiling!

Mistune custom lexers - we are going deeper

2015-12-21T00:00:00+00:00

Mistune is the fastest markdown parser written in pure Python. And it has good documentation. At least so it seemed until I tried to write a custom lexer (hint: tests are a good source of inspiration).

Vocabulary

Even though this is a recipe for solving a specific problem, I’ll start off with some basics and a definition list.

Grammar - set of rules for rewriting strings. In case of Markdown we are rewriting rules like **some string** to some string. Basically I’m saying “if I use two asterisks followed by any random string and end with another 2 asterisks, wrap the string into an HTML <strong> tag”. The full list of those rules is available on Daring Fireball website. A formalized spec of Markdown, a big community effort is also available on CommonMark website. For example, it goes into much bigger detail when and how strong emphasis should be applied.

Inline grammar - rules of the grammar that when rewritten will appear, well, inline. Examples of inline rules in Markdown are links, emphasizes, images, etc.

Block grammar - conversely to inline grammar, these rules of the grammar will take up a whole block. Examples are headers, paragraphs, code blocks, lists, tables, etc. Inline and block grammar rules directly translate to HTML definition of inline vs block.

Lexer - program that is able to convert a series of characters into a sequence of tokens according to the grammar, and it’s doing that using regular expressions.

Renderer - program that controls the output of grammar rules. There is no one fixed way to go from **strong** to strong, but rather an infinite number of ways to adjust that strong output.

Mistune offers a way to have custom renderers and custom lexers. Add whatever specific rules and outputs you want. The example from the docs picks GitHub Wiki links and creates a custom inline renderer to output it as an HTML anchor tag and an inline lexer that defines a regular expression capable of parsing [[Page 2|Page 2]].

Scratching my itch

I want to be able to refer to people by their twitter username and link to their twitter accounts when I’m writing in Markdown right on my blog (this is a theoretical desire, I’m actually not using mistune on this blog yet :sad_face:). This is an inline grammar and it’s going to be composed of @ and a sequence of characters that only accepts letters, numbers and underscore. Replace twitter with any website that has accounts and direct URLs to them.

Basically the thing twitter is already good at: recognizing @username and linking to this account (if it exists).

Following the docs available on mistune README I came up with this:

import re
import mistune


class UsernameRenderer(mistune.Renderer):
    def username_link(self, link, username):
        return '<a href="%s">%s</a>' % (link, username)


class UsernameInlineLexer(mistune.InlineLexer):
    def enable_username_link(self):
        self.rules.username_link = re.compile(r'@(\w+)')
        self.default_rules.insert(3, 'username_link')

    def output_username_link(self, m):
        username = m.group(1)
        link = 'https://twitter.com/%s/' % username
        return self.renderer.username_link(link, username)


if __name__ == '__main__':
    renderer = UsernameRenderer()
    inline = UsernameInlineLexer(renderer)
    inline.enable_username_link()
    markdown = mistune.Markdown(renderer, inline=inline)
    print markdown('Hello @username')
    # <p>Hello @username</p>

Aaaand it doesn’t do its job. It follows the example of the wiki link quite closely: a custom renderer, a custom inline lexer which adds a new rule using a really simply regular expression and then all the initialization. Why doesn’t it work then?

Going deeper

Surprise surprise! It doesn’t even match my regular expression. To find out why I’ve pdb-ed a couple of times into the one-file mistune package and learned the following:

Mistune takes the initial string and runs it through all the rules (by the way the username_link rule is there) to match at least something. It uses re.match(), which specifically is looking for a pattern match at the beginning of the file (for finding a match anywhere within the string re.search() is used). And yes, if we try to do print markdown('@username') it will work as expected and return <a href="https://twitter.com/username/">username</a>.
Once something was matched it will slice the matched bit from the string text = text[len(m.group(0)):] and continue with point 1 until the whole string is exhausted.

What happens in our case is that the string “Hello @username” is treated as a whole paragraph. How come Hello [[Link Text|Wiki Link]] is recognized as a paragraph that also contains a wiki link?

The solution lies in the text pattern of the InlineGrammar class. Look at this text = re.compile(r'^[\s\S]+?(?=[\\<!\[_*`~]|https?://| {2,}\n|$)'). Zoom in (?=[\\<!\[_*`~]|https?://| {2,}\n|$). This is called a lookahead assertion, which matches if the stuff between (?=...) will match, but doesn’t actually consume it. The lookahead assertion contains a couple of characters, which will make mistune slice the string and try to render the token. [ symbol is there, but @ is not. That’s why the wiki link example works just fine and the username link example doesn’t.

Another way

This other way involves overriding the text pattern to take into consideration @ symbol and tokenize the string per our wish. This is my ugly solution (code below is scrollable to the right).

import copy
import re
import mistune


class UsernameInlineGrammar(mistune.InlineGrammar):
    username_link = re.compile(r'^@(\w+)')
    # Override the text grammar pattern to contain the `@` as a stop element in the lookahead assertion
    text = re.compile(r'^[\s\S]+?(?=[\\<!\[_*`~@]|https?://| {2,}\n|$)')


class UsernameUrlInlineLexer(mistune.InlineLexer):
    """Inline lexer for @<username> which links to the user twitter page."""
    default_rules = copy.copy(mistune.InlineLexer.default_rules)
    default_rules.insert(3, 'username_link')
    default_rules.append('text')

    def __init__(self, renderer, rules=None, **kwargs):
        if rules is None:
            rules = UsernameInlineGrammar()

        super(UsernameUrlInlineLexer, self).__init__(renderer, rules, **kwargs)

    def output_username_link(self, m):
        alt = m.group(0)
        username = m.group(1)
        url = 'https://twitter/{}/'.format(username)
        return self.renderer.link(url, False, alt)

if __name__ == '__main__':
    markdown = mistune.Markdown(inline=UsernameUrlInlineLexer)
    print markdown('Hello @username')
    # <p>Hello <a href="https://twitter/username/">@username</a></p>

In this case we are extending the InlineGrammar, adding a new rule and overriding the existing text rule, which isn’t a very pleasant experience. Notice the username_link grammar regular expression has ^ (caret) symbol, which tells it look from the very beginning of the string. Also notice the lack of any custom renderer - I just decided to use the link renderer, which it yield the same result.

Fin

Congratulations! You’ve made it to the end! Got a better solution? Share it with everybody or take a nap :)

DjangoCon, ponies and emoji

2015-06-06T00:00:00+00:00

Early on when DjangoCon Europe 2015 was announced, it instantly differentiated itself from other conferences.

Organized this year in Cardiff, UK, DjangoCon set its goals high by planning 3 days of talks, 2 days of sprints and providing inclusive environment for everybody. It encouraged first time speakers to come and share their thoughts and adventures in Djangoland (© @bmispelon) by launching a speaker mentor program, included a live text transcription throughout the whole conference, offered a free crèche (and we had some lovely young djangonaughts), organized a free open day, which students of Cardiff University benefited from.

In addition to all of this DjangoCon is characterized by an extremely friendly community. Mark said it better:

I can honestly say that the #djangocon community is the nicest, most decent collection of people I have ever met. Breath-takingly lovely ❤️
— Mark Henwood (@mcbhenwood) June 3, 2015

During those few days spent in Cardiff, you realize that Django is not just a framework that we all use and like, it’s the people who make it awesome. It’s also sort of snowball effect. Once you see how Django community treats all the newcomers, you want to share this right attitude with the rest of the world.

DjangoCon did everything right. And it’s pretty hard to get everything right when you have 300+ attendees, volunteer-driven team (actually the most pony-awesome team: Vincent, Baptiste, David, Daniele, Ola, Jason, Geraint) and One Direction invasion.

The tale of DRY with django-crispy-forms | Part II

2015-03-18T00:00:00+00:00

When you write a blog post, people of Internet will point out any mistakes they find. It’s a great opportunity to improve existing code and look into other solutions. After posting The tale of DRY with django-crispy-forms I found out about formulation - a template-based solution for rendering forms, in contrast to crispy-forms that builds the form structure in code.

Also @maraujop – crispy-forms developer – made several very helpful suggestions in the comments area, which brings us to Part II.

Mistakes of the past

The story of Foo people can be found in this archive. Their end was tragic. They knew truth was somewhere close, but they were unable to grasp it to its fullest and failed to survive this cruel software world.

FooForms had their __init__ cluttered with the initialization of a FormHelper object and the attachment of the SubmitCancelFormActions. Nothing bad with SubmitCancelFormActions itself, but as they grew in number overriding the initialization started to look silly.

Decline of Foo marked the birth of powerful Bar people.

Laputa: Castle in the Sky (Ghibli Studio) - How I imagine the fairy tale kingdoms

Reforms

Assume that Bar model, view and template are similar to Foo. Bar people knew they should start with the API, and so they shaped their dreams first.

# fairytale/characters/forms.py
from characters.models import Bar
from common.forms import ModelFormWithHelper
from common.helpers import SubmitCancelFormHelper


class BarForm(ModelFormWithHelper):
    """Model Bar form"""
    class Meta:
        model = Bar
        helper_class = SubmitCancelFormHelper
        helper_cancel_href = "{% url 'some_cancel_url' %}"

What Bar wants is to declare an alternative FormHelper, which is responsible for customizing the form. Also it wants to provide helper attributes, that are different for each form - the cancel URL, for example.

There are 2 unknown creatures here - SubmitCancelFormHelper and ModelFormWithHelper.

SubmitCancelFormHelper as you might have guessed is of FormHelper breed. It tells you right to the face that it appends to the layout submit and cancel buttons. See for yourself.

# fairytale/common/helpers.py
from crispy_forms.bootstrap import FormActions
from crispy_forms.helper import FormHelper, Layout
from crispy_forms.layout import Submit, HTML


class SubmitCancelFormHelper(FormHelper):
    """Custom FormHelper that appends to the layout cancel and submit
    buttons (works only with bootstrap crispy-forms pack). It expects a
    `cancel_href` attribute to be used as cancel button URL.

    Example::

        SubmitCancelFormHelper(self, cancel_href="/some/url/")
    """
    def __init__(self, *args, **kwargs):
        cancel_href = kwargs.pop('cancel_href', '#')
        super(SubmitCancelFormHelper, self).__init__(*args, **kwargs)
        self.layout.append(
            Layout(
                FormActions(
                    HTML("""<a role="button" class="btn btn-default mr4"
                            href="{0}">Cancel</a>""".format(cancel_href)),
                    Submit('save', 'Submit'),
                )
            )
        )

This class expects a cancel_href named parameter. Afterwards it takes advantage of the power it was given by FormHelper and modifies the layout by appending a button that we represent using HTML widget and a Submit widget. SubmitCancelFormHelper serves us well.

What about ModelFormWithHelper mystery. This base class is a bit more tricky, but still nothing out of this world.

# fairytale/common/helpers.py
from django.core.exceptions import ImproperlyConfigured
from django.forms import ModelForm


class ModelFormWithHelper(ModelForm):
    """Custom ModelForm that allows to attach a crispy-forms FormHelper class,
    that will modify in some way the rendering of the layout.

    Example::

        FooForm(ModelFormWithHelper):
            class Meta:
                model = FooModel
                helper_class = FooFormHelper
    """
    def __init__(self, *args, **kwargs):
        super(ModelFormWithHelper, self).__init__(*args, **kwargs)

        if hasattr(self.Meta, "helper_class"):
            helper_class = getattr(self.Meta, "helper_class")
            kwargs = self.get_helper_kwargs()
            self.helper = helper_class(self, **kwargs)
        else:
            raise ImproperlyConfigured(
                "{0} is missing a 'helper_class' meta attribute.".format(
                    self.__class__.__name__))

    def get_helper_kwargs(self):
        """Get all helper attributes from class Meta by stripping them of
        `helper_` part of attribute string

        :return: dict with helper kwargs
        """
        kwargs = {}
        for attr, value in self.Meta.__dict__.items():
            if attr.startswith("helper_") and attr != "helper_class":
                new_attr = attr.split("_", 1)[1]
                kwargs[new_attr] = value
        return kwargs

Ohh, that’s a lot of code. Well, docstrings make it 1/3 of the listing. Let’s see what ModelFormWithHelper is trying to tell us.

In the initialization phase it checks if the class has a helper_class attribute. It’s mandatory to be there, otherwise there’s no point in using this class at all. If there isn’t, ModelFormWithHelper will throw an exception to remind you, Hey, you forgot that attribute - helper_class! Otherwise, it will initialize the form helper class and attach it to self.helper. But before that it extracts the helper kwargs.

Look at the dream BarForm class. Bar people wanted to have a custom cancel URL and used a meta attribute helper_cancel_href. Method get_helper_kwargs() deals with all the helper_ attributes and strips them of that prefix to ultimately feed them to the helper.

The class is not tied to any specific form helper, so you can create as many custom form helpers as your kingdom needs.

That’s it. Now forms will become more compact and readable due to the new meta attributes.

Pagination in Django

2015-01-29T00:00:00+00:00

Isn’t it delightful when things just work? That’s how it is for Django pagination feature.

Say you are writing a blog in Django. Actually seems like a silly idea. If you want a blog, get yourself a static blog, or tumblr, or wordpress, or blogger (is it still popular?). Ok, maybe it’s not a blog. Probably you want to display a list of books in a paginated way. Definitely not cool to list all 2k books on a single page.

I like Django class-based views. They do for you all the hard work if you stick to the basics. So let’s start writing a CBV. Assume the model contains any kind of gibberish.

# my_library/library/views.py
from django.views.generic import ListView

from library.models import Book


class BookListView(ListView):
    model = Book
    paginate_by = 10
    template_name = 'book_list.html'

This is basically what we ought to do to enable pagination. Just specify how many objects we want on one page. Let’s take a look at the route and the template.

# my_library/urls.py
from django.conf.urls import patterns, url

from library.views import BookListView

urlpatterns = patterns('',
    url(r'^books/$', BookListView.as_view(), name='book_list'),
)

{# my_library/templates/book_list.html #}
{# omit all the additional HTML markup and leave the bare minimum #}
{% for book in object_list %}
  {{ book.title }}
{% endfor %}

Now if you start your server and access the URL 127.0.0.1:8000/books/ you will get 10 book objects listed on the first page. In order to access the second page you need to enter 127.0.0.1:8000/books/?page=2. But your user isn’t supposed to know that (don’t even think about writing a manual for that and giving it to the end user). That’s why we will create pagination buttons.

Django ListView is so smart that it will add to your context some special objects that will allow you to build those pagination buttons. In fact the pagination we will build is reusable, so we will put in a separate snippet and include to any template that requires pagination.

{# my_library/templates/snippets/pagination.html #}
{% if is_paginated %}
  <nav>
    <ul class="pagination">
      {% if page_obj.has_previous %}
        <li>
          <a href="?page={{ page_obj.previous_page_number }}">
            <span>Previous</span>
          </a>
        </li>
      {% else %}
        <li class="disabled">
          <a href="#">
            <span>Previous</span>
          </a>
        </li>
      {% endif %}

      {% for page in paginator.page_range %}
        <li {% if page == page_obj.number %}class="active"{% endif %}>
          <a href="?page={{ page }}">{{ page }}</a>
        </li>
      {% endfor %}

      {% if page_obj.has_next %}
        <li>
          <a href="?page={{ page_obj.next_page_number }}">
            <span>Next</span>
          </a>
        </li>
      {% else %}
        <li {% if not page_obj.has_next %}class="disabled"{% endif %}>
          <a href="#">
            <span>Next</span>
          </a>
        </li>
      {% endif %}
    </ul>
  </nav>
{% endif %}

Now back to our book_list.html template.

{# my_library/templates/book_list.html #}
{% for book in object_list %}
  {{ book.title }}
{% endfor %}

{% include 'snippets/pagination.html' %}

Dead simple. And this snippet will serve you well for all kinds of list views.

Take me to Bali

2015-01-22T00:00:00+00:00

Bali is probably one of those beautiful and exotic places, that are actively advertised at your local travel agency. Paradise on earth, I think, they call it. Not without a reason, of course. It’s this beautiful, haunting place, where you imagine yourself lying on the white-sand beaches, sipping coconut cocktails and watching the azure ocean. You know, like in movies.

Selamat Datang

Welcome to Bali! You and the other 3 million curious tourists annually made it to the destination. Seriously, Bali has almost no off season. People come and go all the time. According to the statistics, provided by Bali Government Tourism Office, the island hosted on average 300,000 newcomers each month during 2014.

Tropical Plumeria

Even though Indonesia has 18,307 islands, Bali stays the top one attraction. So be prepared to meet crowds of people. To your attention, Visa on Arrival (VOA) costs 35 US dollars and don’t forget to spare 20,000 IDR for the departure tax. Generally prices grow every year, so get informed before the trip.

Where to go

When I was hearing Bali, I always thought of just Bali. It never crossed my mind that the island is big enough to have a variety of touristic flavors. North is different from South. Center of the island is different from its outer parts. Since I have spent a month and a half in Indonesia, I got plenty of time to try 3 types of Bali life.

Kuta is a famous beach on Bali. It’s full of life, since most of the Australians stop here. As a matter of fact, Australians represent the major nationality visiting Bali. It is a 3-hour flight on average from Darwin to a paradise-like, cheap and always welcoming location.

Kuta Beach

Kuta was very attractive to me, because it has the best beginners surfing beach. The ocean surface by the beach is sandy with almost no rocks, it is not deep and the waves are good both for beginners, as well as for pros. Kuta doesn’t have the perfect tube wave, but I guess I am far from mastering that one.

There isn’t much you can do around Kuta, except for party and surfing. If you can get out of the village, then you can visit Pandawa Beach. The water is clean and still. There are less people and the cliffs around look lovely.

Clean and still waters at Pandawa Beach

Cliffs at Pandawa Beach

Unfortunately Kuta is the most annoying place on all Bali. Every time you walk past local people, they will try to sell you goods or services: massage, motorbike rent, taxi, tattoo, souvenirs, board rent, surfing lessons. They never get tired of asking you again and again.

I guess I sound very picky, but in 2 or 3 days most of the tourists start to feel the need to avoid the eye contact with vendors. On the other hand the behavior is understandable, since the competition is huge. The advertising is aggressive and pushy, but it works for the most part.

Some of the marketing strategies are more sophisticated. Locals try to establish a relationship with you. Make a little smalltalk. Lure you with their fake interest. And when time is right, they tell you what they really want - sell stuff.

Sunset at Buta Beach

Most people who tend to seek peace and balance go to Ubud. Ubud is a lovely place, 2 hour ride from Kuta to the north. It breaths culture and style. It can be summarized as art, yoga and dance. Ubud has several museums and plenty of galleries.

There are a lot of home-stays in Ubud, so it is kinda easy to find intimate accommodation for a low price. Search for a home-stay that offers breakfast. They cook tasty pancakes and always include fresh fruits - yummy.

Ubud has this big hype around organic stuff: organic food, organic drinks, organic yoga pants. I might be wrong, but I assume the local cheap food uses same veggies as organic restaurant do. It’s just organic rises the price tag.

There are some famous foot tracks in Ubud, which lead you to beautiful fields and hills. I have really enjoyed the Campuan Ridge walking tour.

If you like monkeys, go to Monkey Forest. I am not sure who is bothering who: monkeys the tourists or tourists the monkeys. I have seen people who were not respecting the rules of the forest and in some way harming the animals. Monkeys aren’t really polite as well. They will snatch shiny objects and will climb on to you if they think you have food.

Monkey Forest, Ubud

Up to the north is tranquility. We visited Lovina Beach - a real backpackers heaven. There are considerably fewer people and the accommodation prices can be very low. Nearby is the Bali sea, mountains and beautiful nature.

Postcard view, Lovina Beach

Lovina is known for its volcanic black sand beaches. If you go swimming, there are barely any waves. It’s an amazing feeling to swim in quiet rain and witness some gorgeous rainbows (and double rainbows).

It is a fishing village, so expect to see a lot of boats. But the real symbol of Lovina are the dolphins. By the beach there is a tall status of a dolphin. They come to Lovina for the fish and no one loses the opportunity to see the mighty creatures. Wake up at 6 in the morning to catch a boat to see the wild dolphin chase. It’s kinda funny how tens of boats try to guess where exactly the dolphins will appear. And when they do, the whole gang of roaring boats tries to catch up with them.

Boats and fishing in Lovina Beach

A wild dolphin chase

It’s worth going to the north to do snorkeling and diving to the Menjangan Island. I will never forget how impressed I was by the rush of the tropic aquatic life near the riff. Those pretty little fish going round their own business, as if never really bothered by weird guys in masks.

Beautiful Menjangan island

On the way to Menjangan island

Temples

For the most part Indonesia is a Muslim country. But Bali is different. Bali practices Hinduism as religion. And you could feel it even with the soles of your feet. It is a big deal in the life of locals. They wake up in the morning and prepare the offerings for the God, light up a scent stick and repeat the process by the afternoon and evening. It doesn’t seem like a burden to them, they feel happy and accomplished to perform the ritual daily.

Offerings prepared 3 times a day

That said, Bali is overwhelmed with temples. To be honest, most of the them look the same with slight differences. There are important temples, then big touristic temples and everyday temples. Important ones are of some big significance for local people, but on the exterior they have nothing special. Pura Besakih is the largest template of Bali and was my one and only saint place, which I visited. And then come the everyday temples - the ones built by the house. Yes, more or less every house has a temple of its own.

Puka Besakih - largest temple on Bali

If you plan to visit more than one template, buy a sarong up front. Men and women are not allowed without one. Besides even if you get inside the template, you actually just walk around it, since visitors are not allowed in the prayer/meditation zone.

Nature

Mesmerizing tropical nature is what left me speechless. During the trip good and bad stuff happened, but I always got reconciled somehow by the natural beauty of the island.

Bali is famous for its wonderful rice fields. The green cascading pieces of land are protected by UNESCO, otherwise endangering the main Bali cultivation to be torn apart by greedy construction companies.

Rice Fields, Bali

Green Fields, Bali

Stairway to rice fields, Ubud

Bali has to offer several hot springs mostly found at the north of the island. The way to Air Panas Banjar Hot String is somewhat famous, for the road being beautifully shadowed by old trees, with a clear blue coastline by the side of the road, which makes up for nice bike ride.

On the way to Air Panas Banjar, Lovina Beach

Air Panas Banjar, Hot Strings of Banjar, Bali

Volcanoes - just some mountains with some lava boiling in it. But it’s kinda impressive to see a real active smoking volcano. We’ve been to Batur Volcano on motorbikes and the whole trip was a blast.

Batur Volcano

Road to Batur Volcano

Batur Volcano ashes

That being said, there ain’t no sunshine without rain. Unfortunately ecology is not something locals care about. People throw tons of plastic into the ocean and guess what, during the rain season it all comes back to the shore. The amazing tingling of plastic bags in the water - a dream come true.

Garbage by the riverside

Kuta Beach during rain season

Fruits

That smell can never be forgotten - durian. Druin is a fruit extremely liked by the Asian population. There is very little point in trying to describe the smell, but as they say it, whether you love or you hate it. I got the last one.

Durian fruit

Along with durian there are all kinds of fruits: passion fruit, dragon fruit, bananas, papaya, pineapple, watermelon, lychee, mango, etc.

Pitahaya or Dragon fruit

But the real treasure is Manggis. This fruit is not particularly rich in nutritional value, but it’s inner juicy vesicles are very tasty. The fruit is in high demand always. Manggis season is short and hence big price fluctuations happen. It’s nice when you are able to catch the season.

Mangosteen (Manggis) - the Queen of Fruit

Locals

You walk by and every local person will greet with “Hi” or “Hello”. Local greetings sound the same as English versions by the way. The hospitality of the place is felt everywhere as long as you behave. Once you crossed the line, bad things can happen. Don’t forget that the police is corrupt, so no easy way to get out of a bad situation.

The whole time we were on Bali nothing was stolen or attempted to be stolen. At the inns they usually don’t have safe boxes, but it felt like you don’t even need them. I was told by the locals that indeed theft doesn’t occur often and for the most part it happens to drunk tourists late in the night returning from a party.

Balinese people are very friendly and smiley. They will always help you if you have the intention to purchase something from them. But with all the welcoming appearance, it’s never a bad idea to be prudent. I hate to say it, but Balinese will lie and deceive you if they see profit. For example, you might be deceived at some local money exchange booths ending up with less money than you’d expect. Nevertheless nothing seriously bad ever happened.

Bargaining is more than welcome. Once the price was dropped by 70% and I didn’t say a word. To me all the prices felt dummy afterwards, since it’s rather hard to estimate the real value. Nevertheless white people will overpay in any case. You can’t convince them to sell goods by the local real price. It just never happens.

Food

I felt in love with Balinese food. It’s slightly spicy, averagely salty and delicious for my tastes. Bali has a lot of vegetarian and vegan dishes. Generally meat is more expensive, seafood is mainly available and vegetables and rice are everywhere. Most common dishes are mie goreng (fried noodles), nasi goreng (fried rice), nasi campur (rice with toppings like meat, veggies, souses), soto bakso (soup with meatballs), soto ayam (chicken soup). My all time favorite is the mie soto - chicken soup with noodles, prepared at a small local dinner in Ubud. I can fill my mouth all watery now just by the thought of mie soto.

Mie goreng take-away in banana leaf

Rice is a complement to every meal. Even McDonals and KFC will serve you rice.

As in most Asian countries food is everywhere, and particularly on the street. The authentic bakso is sold by guys driving a bike-tiny-kitchen assembly. They stop where there are customers and fix you a fine bakso, that you eat on the spot.

Of course hygiene should be of concern and you would probably prefer a more “posh” place for dining. In reality there is one universal rule: if locals eat that food, it’s safe for you too. To be mentioned, I have never had food poisoning while on Bali.

Transport and traffic

Traffic is big and hectic. Few rules apply. For the most part just watch out! People drive bikes, all kinds of bikes. It’s a like a big collection of cute Vespas, Scoopies and not so cute Honda Vario. It’s easy and cheap to rent a bike, if you like extreme driving. Keep in mind that cars are not exactly respectful towards the bike drivers and traffic jams are awful.

Vespa bike

Activities

It won’t be boring on Bali, trust me. Locals are offering plenty of activities.

Start with lying on the beach with a good book and get a gorgeous tan. Catch the wave on a surfing board, fall, try again. Go trekking on the top of a volcano. Get a bicycle and visit some nearby villages, buy bakso for it’s real price of 5,000 IDR. Soak in the greenish water of a hot spring. Don’t forget about white water rafting in Ayung River - instructors make it an unbelievably fun experience. Take long walk to discover the blooming nature of Bali. Go to a cooking class and learn some secrets about Bali cuisine. Explore the templates if you feel attracted to the spiritual side of the island. Get fit by taking yoga or traditional dance classes or relax by having an oily Balinese massage. Go diving or snorkeling to special beautiful spots. Join a group of tourists to see the Legong and Kecak dance. I think there is a lot more to this. Once you are there, you will figure it out.

Goodbye

One last piece of advice: Live in Bali once, but leave it before it makes you soft. And that’s what we did.

Contributing to open source like a boss

2015-01-11T00:00:00+00:00

This is a collection of the most obvious, nevertheless overlooked, bullet points that every contributor should be aware of when making the first contribution to an open source project.

I remember my first contribution to MoinMoin Wiki Engine and it was terrible. And I’m not talking about code (which was probably also pretty bad), but about all the surrounding elements to an open source contribution. With time you learn how to do it the right the way.

Now I find myself in the position of the reviewer and I see tons of mistakes.

But it’s OK. I was a noob quite recently and for some projects I am still holding on to my noob award. Everybody passed through the stage of first contribution. But once you learn and get comfortable, it’s very false to start assuming that newcomers should catch on very fast. I think newcomers need help and guidance. That’s why I figured it’s a good idea to disseminate some of the common rules of open source world.

How do I get started?

If you made the decision to contribute, don’t go to the organization’s communication channel and say that out loud. That’s not bad, it’s just useless. Long term contributors and maintainers are like smart girls dating teenage boys: they ignore everything boys say and only pay attention to what they do.

And since you set your mind to contribute, take some time to search for project documentation and read it. The question How do I get started? is mostly irrelevant. I find it kind when people on the channel point out to relevant links and give short instructions. But… come on! Open source projects’ documentation is not a hidden treasure chest, it’s out there indexed by Google. Don’t be lazy.

Spend at least an hour trying to find relevant developer documentation. It’s perfectly OK to ask for help if the docs are missing, lacking or misleading. In fact, if you figured the documentation is not straight-forward, express your concern and propose a solution. Your feedback will be appreciated if the community is welcoming.

Follow that guide

Usually a community that cares about code quality will comply to a code style guide. This is a very important document, read it carefully. It will teach you about naming conventions, do’s and don’ts and other recommendations. For example, in Python world a lot of projects follow PEP8 - if you haven’t seen a style guide, here is what it looks like.

Pay attention to the tools the community uses. They might prefer one pastebin over another, have a community-wide wiki, expect people to use a single code review tool. Follow those recommendations. It makes everybody’s life easier.

Find easy TODO

Projects usually have a bug tracker, whether it’s a Trac tracker, GitHub issues or simple wiki page. Doesn’t matter as long as the tracker is up-to-date. Try to see if they have tickets with easy tag or something similar. Search for the one you feel most confident about.

If you found a bug while using the software, check if it’s being already listed. If not, report the problem.

Further express your willingness to work on the ticket: assign it to yourself or leave a comment - whatever works best. The reason is nobody wants duplicate work.

Well, now is your time to shine.

Branch that out

Most probably you will need to create a new feature branch. Holds true for git, somewhat for mercurial. Name the branch appropriately. Don’t name it stephen or my_branch.

Check what is the leading branch for development, might be master or maybe dev. Create the feature branch from the leading development branch. Pull the latests changes. If the branch was updated while you were working on your feature branch, rebase your work. It’s so much easier to merge the contributions of others if their branches contain the latest changes.

Do this for every feature. It might seem like overhead, but it actually saves some pain in the future.

Run those tests

After you solved the ticket, take some time to run all the necessary scripts before committing. Those might be unittests, UI tests, linters. Pay attention to their output. If the output makes little sense to you, take a step back and read about the tool to figure out the meaning of the output.

It’s pretty cool when a project uses a CI (Continuous Integration) tool. It tells the developers straight away if something is wrong. Thought that doesn’t spare you and others from submitting clean working code.

Seriously, run those tests.

Code review

Go back to the docs and search for the section that tells you where and how to submit code for review.

There are plenty of code review tools: Gerrit , Rietveld, Phabricator, GitHub pull requests. So there must be one that community prefers. Before you put your code on review, check if you are submitting only the relevant pieces, remove all the debug code, don’t submit sensible information (passwords, keys) or local user configuration.

Your code review should contain a title and a description if necessary. Don’t name it First contribution or My first pull request please review asap.

And once you did submit the code, go to that link and review it yourself. It’s hard to emphasize how important is that step. Take a minute or two to scan with your eyes all the code. You will be surprised to find out that you have slipped some unwanted code in there or you just found a better solution.

Pull request

Pull request process can be merged with the code review process. Contributors can review your code once you make the pull request. Or they might be separate. Matter of preference.

Repeating myself, but clean title, description, commit messages are important. Refer to the ticket in commit message or pull request title if it’s necessary.

Limit a pull request to one to 3 commits. Don’t overload code reviews and pull requests with tons of code. It’s almost impossible to review big chunks of code. A method, a function, a small class, some bugfixing - OK. Class with 10 methods, extra 5 templates with controllers and routes - not OK.

Reviewing isn’t easy. It takes quite some time and the reviewer holds same responsibility for your code as you do, since he or she decides to accept or discard your contribution.

Keep it short. Keep it simple. And people will love it.

Wrap-up

It’s OK to make mistakes, as long as we acknowledge and fix them. The rules we have imposed on ourselves help keep the codebase readable and maintainable, hence ensure the longevity and popularity of the project. Comply to this set of rules and, boy, you’re gonna become a pleasant person to work with.

Open source is a good place to learn from others. Don’t miss that opportunity. Appreciate the feedback.

Systers Portal

2014-11-11T00:00:00+00:00

Today I was listening to Swift episode by Programming Throwdown podcast series. At some point hosts raise the question of open source involvement. You probably also received this piece of advice:

If you want to get your hands on some programming as a beginner or a recent graduate, try doing some open source.

That is rather vague. What most people skip saying is what kind of open source work should you start with. Hosts of the show point out the fact that it’s not necessary to jump right into big popular projects, like Linux Kernel or PostgreSQL, simply because it will take tons of time to read and understand the existing codebase and get your patch accepted by the core maintainers. So the solution is to seek for a small open source project, that involves a technology you like or want to get better at it, preferably with few developers involved.

Systers Portal is one of those one-man-show projects, that might suit very well the open source beginners.

What is that?

Systers is the world’s largest email community of women in technical roles in computing. It is indeed huge and covers a lot of technical areas. It started as a small mailing list and grew to 4,000 members from all over the world. Considering the modern needs, a single mailing list isn’t very nifty to manage such a huge community.

Hence the idea of a portal came up. Systers Portal is a unified platform for Systers and its sub-groups to share information and get the latest news. This is a place to manage sub-communities, latest news, interesting and useful resources, community members.

Systers Portal was started in summer 2014 as a Google Summer of Code project involving two students: me and Chitra Khatwani, under the mentorship of Lynn Root, Rosario Robinson, Promita Bose and Laura Cassell. We launched a prototype, hosted on portal.systers.org.

Disclaimer: the prototype hosted on portal.systers.org isn’t going to match exactly your development version from the official repository. The development version incrementally adds features present in the prototype.

Reasons

Here are some of the obvious reasons why Systers Portal is so good for you:

It is a Django project, build using best practices and techniques, with readability and maintainability in mind. We try to keep up with the latest releases, hence we are using Django 1.7 with a set of up-to-date packages.
We are strict about styleguides and code readability. You will learn fast about one of my favorite Python mantras, called PEP8.
We have 100% code coverage and care about unittests.
We are hosted on GitHub, ready to be forked and accepting pull requests.
We have documentation. Particularly documentation how to get started, what should you know before getting down to code, how to run the project locally. The First Contribution article will drive you through all the necessary steps to make your first contribution to Systers Portal. The code contains helpful docstrings as well.
There is currently one developer working on Portal - it’s me. It is generally easier to read a single mind expressed in code, than a whole bunch. This doesn’t imply that the code is always obvious and clear. Particularly every time I review old pieces of code, I rewrite something. I need your help to see the errors and solutions.
We have a list of easy TODOs (initially created for Google Code-in).
By contributing to Systers Portal, you will help women from technical fields to enjoy a friendly environment via a unified portal.

Links! Links! Links!

GitHub repo - github.com/systers/portal
Documentation - systers-portal.readthedocs.org
Systers Portal prototype - portal.systers.org
About Systers - systers.org

The tale of DRY with django-crispy-forms

2014-11-04T00:00:00+00:00

So you have this Django app. If you are like me and don’t have quite the skills for a good UI design, you probably stick to bootstrap. You know you will have a bunch of forms and you don’t want to style them yourself. Let’s say, you came to the idea that django-crispy-forms fits your needs perfectly (it definitely fits mine).

How it all began

This is a story about a model Foo.

# fairytale/characters/models.py
from django.db import models


class Foo(models.Model):
    """A magical creature from Foo dynasty"""
    mighty_name = models.CharField(max_length=255)
    kingdoms_count = models.PositiveIntegerField(default=0)
    email = models.EmailField()  # What? Magical creatures also have emails.

Model Foo was lonely, so it summoned a view, a template and a url.

# fairytale/characters/views.py
from django.views.generic.edit import CreateView

from characters.models import Foo


class CreateFooView(CreateView):
    """A create view for Foo model"""
    template_name = "characters/create_foo.html"
    model = Foo

# fairytale/templates/characters/create_foo.html
{% extends "base.html" %}

{% load crispy_forms_tags %}
{% block content %}
  <div class="row">
    <div class="col-md-6">
      <h1>Create Foo</h1>
      <hr/>
      <div class="well">
        {% crispy form %}
      </div>
    </div>
  </div>
{% endblock %}

The base.html contents are left out, as they include standard HTML markup, bootstrap stylesheet include and a mandatory template block content (in our specific case).

# fairytale/fairytale/urls.py
from django.conf.urls import patterns, url

from characters.views import CreateFooView


urlpatterns = patterns(
    '',
    url(r'^foo/create/$', CreateFooView.as_view(), name="create_foo"),
)

And this is what we created so far:

Such cool, right?

Oh, wait! Where are the buttons? How do I submit this thing? Foo the Emperor must live!

Ok, I will need a Cancel anchor tag that will look like a button, because I want to redirect to some other page, and a Submit input. Easy!

# fairytale/characters/forms.py
from django import forms
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit, HTML
from crispy_forms.bootstrap import FormActions

from characters.models import Foo


class FooForm(forms.ModelForm):
    """Model Foo form"""
    class Meta:
        model = Foo

    def __init__(self, *args, **kwargs):
        super(FooForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper(self)
        self.helper.layout.append(
            FormActions(
                HTML("""<a role="button" class="btn btn-default"
                        href="{% url "some_cancel_url" %}">Cancel</a>"""),
                Submit('save', 'Submit'),
        ))

Meh, not the prettiest way to achieve what I want, but apparently this is the only easy and straight-forward way. Also don’t forget to update the view.

# fairytale/characters/views.py
ifrom django.views.generic.edit import CreateView

from characters.models import Foo


class CreateFooView(CreateView):
    """A create view for Foo model"""
    template_name = "characters/create_foo.html"
    form_class = FooForm  # the change is over here
    model = Foo

And so this is what we get:

The lack of space between the anchor tag and the input looks awful, right? This tiny cosmetic issue was discussed in issue #62 and solved for groups of buttons and inputs. Unfortunately it persists in case of an input/button and anchor tag with role="button". If it’s bugging you, add some margin by yourself.

The misfortune

Life goes on and Foo objects seem to be doing just fine. Until a great misfortune takes over their destiny. It doesn’t seem like a big deal, since all that happened was the addition of two more characters: Bar and Baz. What harm can they potentially cause? Well, let’s see.

Bar and Baz follow the same old path Foo did a long time ago: model/form/view/template/url. The form again will initialize a FormHelper to append those 2 elements. It seems like every form will need them (like duhh, how can I get around the form without them?). And so we copy-pasty, we copy-paste.

And then the pressure of the invisible monster can be felt, crawling under the skin and whispering to you: “You will do this over and over again. For every form. And when you will want to change a tiny detail, you will change it for every form, because you want to stay consistent. That can be the change in the bootstrap anchor tag interface, class of the Submit input, text of the anchor tag.”

This is the punishment for the violation of DRY (Don’t Repeat Yourself) principle.

It’s not too late to fix the situation, until the quicksand doesn’t devour all the codebase into the abyss of “refactoring is too expensive for us”.

Layout objects to the rescue

At first, it seems like a good idea to use Composing layouts. We take that chunk of layout and simply store it into a variable somewhere. But how can we change that Cancel URL? Or maybe we plan to have different input values for submit input for different forms. There might be several variables that we would like to keep flexible for a more or less fixed piece of layout.

One of the solutions is to create our own custom ObjectLayout. It involves a little bit of django-crispy-forms internals, but don’t worry, I’m here to help.

This is where we start from - creating your own layout objects. As it says in the current section, we investigate the objects that live in layout.py and bootstrap.py. We come up with the following cure for the misfortune:

# fairytale/common/crispy_forms/bootstrap.py
from crispy_forms.bootstrap import FormActions
from crispy_forms.layout import LayoutObject, Submit, HTML


class SubmitCancelFormActions(LayoutObject):
    """Custom bootstrap layout object. It wraps a Cancel anchor tag and a Submit
    input field in a <div class="form-actions">.

    Example::

        SubmitCancelFormActions(cancel_href="/some/url/")
    """
    def __init__(self, *fields, **kwargs):
        self.cancel_href = kwargs.pop('cancel_href', '#')

    def render(self, form, form_style, context):
        layout_object = FormActions(
            HTML("""<a role="button" class="btn btn-default" href="{0}">
                    Cancel</a>""".format(self.cancel_href)),
            Submit('save', 'Submit'),
        )
        return layout_object.render(form, form_style, context)

In the __init__ we extract all the arguments we are interested in, for example the URL of the Cancel anchor tag. The render method produces the HTML string with all the nodes translated. There is no need to create our own template to be rendered, since we can reuse render method provided by the django-crispy-forms. It already knows how to translate all those objects we have put together, to plain HTML.

# fairytale/characters/forms.py
from django import forms
from crispy_forms.helper import FormHelper

from characters.models import Foo
from common.crispy_forms.bootstrap import SubmitCancelFormActions


class FooForm(forms.ModelForm):
    """Model Foo form"""
    class Meta:
        model = Foo

    def __init__(self, *args, **kwargs):
        super(FooForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper(self)
        self.helper.layout.append(
            SubmitCancelFormActions(
                cancel_href="{% url 'some_cancel_url' %}")
        ))

Maybe it doesn’t seem like a big deal, saves us maybe 2 lines of code in the form and adds up lots of code some place else. As the monster warned us, the real danger comes with the growth of objects and forms in number. At some point it will become much more complicated to make any changes to those form actions. It is a small enhancement that might save time for future me.

Alright, that looks much better. Both for Foo, Bar and Baz. Oh, and don’t forget about unittest, otherwise another monster will come to bite your ass.

Let’s see how can we write a unittest for our little layout object. I will cheat a bit and use some test tools and shortcuts created specifically for django-crispy-forms.

# fairytale/common/tests.py
from django.template import loader, Context
from django.test import TestCase
from crispy_forms.tests.forms import TestForm
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Layout

from common.forms.bootstrap import SubmitCancelFormActions


class FormLayoutObjectsTestCase(TestCase):
    def test_submit_cancel_form_actions(self):
        """Test custom SubmitCancelFormActions bootstrap layout object"""
        template = loader.get_template_from_string(u"""
            {% load crispy_forms_tags %}
            {% crispy form %}
        """)

        test_form = TestForm()
        test_form.helper = FormHelper()
        test_form.helper.layout = Layout(
            SubmitCancelFormActions()
        )

        c = Context({'form': test_form})

        html = template.render(c)
        self.assertEqual(html.count('class="form-actions'), 1)
        self.assertEqual(html.count('role="button"'), 1)
        self.assertEqual(html.count('href="#"'), 1)
        self.assertEqual(html.count('Cancel'), 1)
        self.assertEqual(html.count('Submit'), 1)

        test_form.helper.layout = Layout(
            SubmitCancelFormActions(cancel_href="/some/url/")
        )
        c = Context({'form': test_form})

        html = template.render(c)
        self.assertEqual(html.count('href="/some/url/'), 1)

In the unittest we render the form with the custom layout object SubmitCancelFormActions and check for some keywords if present in the rendered HTML.

Happy ending

That’s all folks! Keep DRY and write unittests!

P.S. In case you are still in doubt about the provided solution, here is what Daniel Greenfeld says: “As one of the leads of django-crispy-forms, I approve this blog post. :-)”

See The tale of DRY with django-crispy-forms | Part II for a better solution, if you wanna to go an extra mile.

Erasmus FAQ

2014-10-27T00:00:00+00:00

I see a lot of interest from students towards going on Erasmus. Their excitement is understandable, nevertheless students tend to rely heavily on other’s experience and ask same questions all over again (rather than reading piles of information on official websites). Hence I have decided to collect all those questions by creating this FAQ. The questions are not grouped anyhow and are not in any specific order.

1. How many credits are required to finish one semester of Erasmus program?

In the official “Learning Agreement” you will see that each student is expected to collect 30 credits per semester. In reality you can have less, but still a fair amount of courses and credits. At the Czech Technical University we were given the following numbers:

more than 27 – you got the perfect score
23 - 27 – that’s a good result
17 - 23 – fair, but you could have done better
less than 17 – you might get kicked out of the program

The numbers are extracted from memory, so expect some delta error. Don’t risk and try to get all the credits you can.

2. What is the case if you are on Erasmus during your last semester? Can you include the thesis credits as well?

I believe this is something university specific, but I can tell about my experience at Czech Technical University. The credits are bound to courses, therefore there are some “Bachelor/Master Thesis” courses. Before being able to take the course, you, as a student, should find a mentor – the guide-teacher that will help you with your thesis and will give you the credits by the end of the semester. Considering this, it is a good idea to spend one semester before your last thesis-semester at the host university. If you have the topic or the field of study you want to work in, it will be easy to spot the right mentor. Approach the teacher and explain what you are expecting from him as a mentor, what are your responsibilities towards the home and host university. Most probably it will work out ok.

3. Do you need to find mentor before going or you can do so later?

It is perfectly ok to find the mentor on the spot. Nobody expects you to email dozens of teachers and find a mentor blindfolded.

4. Where to start research on your options?

I know at least 2 resources - EWENT and ACTIVE. I’ve got not idea why there are 2 programs at the same time and what is the difference between them. Both websites contains clear information about the partner universities, coordinators, courses, scholarships. It will take probably an hour to read everything.

5. In order to become an exchange student, my home university should have parter relationships with other universities (the ones listed on the website)? Are those the only parter universities?

Yes. You can go on Erasmus only to the universities listed on website. There are 7-8 partner European universities (depending on the program) and if any other join the program, they will be listed there most probably.

6. Is it necessary that Technical University of Moldova endorses me?

No. Home university will provide a certificate to confirm the fact that you are their student. And that’s all.

7. With a average mark of almost 9, do I have any chances?

It’s hard to know what are the selection criteria. I might assume that university record is important, along with extracurricular activities, involvement in a variety of projects and a good motivational letter.

8. So how was your Erasmus?

It is definitely a good experience. I have plenty of good memories. I have managed to make good friends, live in one of the most beautiful cities of Europe, see student-friendly environment created to facilitate studies and teamwork.

At times it was not easy, due to the temptation of traveling and exploring, instead of studying. But it all can be balanced well with the proper discipline.

9. Did you write your thesis at host university?

Yes. The core part. Some sections, bugfixes, formatting were done by the end of the semester, when I came back to Moldova to prepare to defend the thesis.

10. How did you find the study program compared to the one from Technical University of Moldova? Did the complexity level grow exponentially?

I would say the study programs are comparable. I didn’t find any difficulty to pass my courses. But I know some of the courses that are extremely complicated to pass at Czech Technical University and require a lot of hard work. If you find the course interesting and important to you nevertheless, you can give it a try at least.

I met some excellent professors at Technical University of Moldova. I was less inspired by Czech Technical University professors. But it’s understandable, since I have spend there only one year and generally, teachers invest less effort in Erasmus students than in regular students.

11. Do I need TOEFL? Or is it ok to just pass an exam at the university?

In case TOEFL is required, it will be stated so in host university information. For example, Dublin Institude of Technology accepts applications only from students who have TOEFL certificate.

I have no information about the English university exam though. Since I was doing my studies at FAF (English Taught Honors Program in “Computer Science”), we were given automatically a certificate from TUM.

12. Can you choose where (college/university) to apply or are you distributed?

You can choose up to 3 partner universities, where you would like to go. The motivational letter respectively should be addressed to all of them. Eventually you will be selected to one of the universities you have listed.

13. Can you go to PhD directly or you need to do Masters first?

If you can convince our home university to get enrolled into a PhD program without graduate studies, then yes. But generally that is very unlikely. According to study specifications, one should have the prerequisites, before going any further (Bachelor – Masters – PhD).

This is it. I hope I cleared out some questions for you. Next application process starts on November 1st 2014, so start your application now and good luck :)

Developers at Eco Village

2014-09-21T00:00:00+00:00

Eco Village supports the idea that sustainable lifestyle can be revived in rural area of Moldova with the effort of the community. It’s also an NGO and a great bunch of people.

It’s pretty cool how people with strong vision can make things happen. At the moment on the edge of Rascova village a big construction site is in action. There are regular workers and volunteers. There is a in-progress building for cohousing (concept that describes how groups of people decide to live next to each other, sharing some of the common space), training center for local community, meditation center and other adjacent constructions. Along with the Eco Village, the founders of the project are offering a house for volunteers, who want to stay more than one day.

I have found out about Eco Village accidentally, right during the time I was searching for a rent in a village. This is not an easy task to accomplish, since apparently nobody thinks that some youngsters would like to come over. Therefore there are no postings about renting property outside of the urban area.

And so 2 developers took their backpacks and winter coats in early September to go to Eco Village. We have spent 4 days being welcomed by the hosts of the volunteer house. The everyday routine included biking, some work on site and coding. During those days we were interested in finding the perfect combination for building earth blocks. Earth blocks are a mixture of clay, soil and other supplements, compressed and left to dry. They are a pretty nice alternative to bricks, earth blocks being environmentally friendly, non-toxic and much cheaper. I imagine getting back after a year and seeing what was build out of those.

Overall it was nice to get out from the city for a while and find a peaceful place to stay. We are thankful for the people who gave us the opportunity and are wishing good luck to Eco Village

Python Moldova Meetup #3

2014-09-08T00:00:00+00:00

Quite recently we organized our third Python Moldova Meetup. We also have updated the official Python wiki page and added Python Moldova to the list. Meanwhile Python Moldova Community exists over a year and counts 154 members at the moment. Though the number of members vaguely defines the Python involvement rate in Moldova.

We didn’t grow yet to give any pseudo-official talks with prepared presentation and extra 5 minutes for questions from the audience. At the moment it’s easier to get people out for a drink in a pub. We were lucky to find a quiet public place during the weekend, where we could discuss different Python-related topics.

At first, we debated over pros and cons of adapting Python as the main programming language for Informatics course in schools. Apparently not every country has Informatics as a mandatory course in their curriculum. Moldova does. So far we had Basic Pascal. From a specific point of view, it might not be necessary to change the language – learning programming is not about the syntax. But… but… it’s so trendy to have Python in schools. In the end we compiled a list of advantages to use Python as introductory language in schools: REPL, clean syntax, indentation as language feature, good standard library, online IDEs for Python, gaming libraries. I assume the decision will be taken in about a year. I would like to see kids playing around with Python in schools.

We did our small improvised Python Module of the Week with an accent on “tools that changed my life”, agreed on the fact that there are no popular podcasts about Python, talked about females in programming world. The gender question in CS is always acute due to the difficulty of balancing between encouraging females and not discouraging males. None of the present males at the meeting, who were the big majority, expressed any discomfort or annoyance while working with females in IT.

Overall it was a nice outing with smart people. To everybody who joined us, I am glad you were there. To everybody who didn’t, you owe us a presentation for the next meetup :P

P.S. All pictures are kindly provided by Ilona Sergheevna, reporter and wannabe programmer who joined us at Python Moldova Meetup.

Buda and Pest

2014-07-05T00:00:00+00:00

Every time going on vacation to some new place I am telling myself to not look into any guides and follow my intuition. I want to be amazed and surprised by what I see and preferable have zero expectations. That doesn’t work always, because you start getting this feeling that you are simply passing by old buildings, monuments, squares and having no idea what and why they are here. A piece of history or a legend helps to grasp the beauty of the place. Too bad for my trip to Budapest I didn’t research on any of that. At least kept my WOW moment.

After a sever-hour drive from Prague, we are finally in Budapest – considered by many of the most delightful cities in Europe. Mainstream is so cool that I couldn’t resist the urge to see it myself. I lied, it just happened that we ended up on our weekend in Budapest.

Baths

I usually say don’t go where all the tourists go, but not in this case. Natural thermal baths are simply amazing. As if you are swimming and having fun in a museum. We have been to Szechenyi. Wikipedia says it’s the largest medicinal bath in Europe – good to know. Szechenyi has outdoor and indoor pools, saunas, steam rooms, whirlpool, streams to massage your neck, feet and ass. Probably there is much more, but I didn’t bother hoping from one pool to another, so I simply enjoyed sunbathing outside.

Landmarks

Be a tourist till the end and go take tons of pictures by the Budapest Parliament. This building is huge, I mean it. I haven’t been inside, but was told very little can be seen and is not worth the money. Don’t forget to check on the monuments around, Museum of Ethnography, a bunch of statues and the rocky beach by the Danube river.

Across the river you can see the Buda Castle, Matthias Church and Fisherman’s Bastion.

Not far from Szechenyi baths you can see Heroes’ Square, the Timewheel and Vajdahunyad Castle. Pack them up together in one sightseeing tour. The story with Vajdahunyad Castle is curious – it was built to celebrate 1000 years since the Hungarian conquest of the Carpathian Basin. Was made initially in cardboard and wood, afterwards being rebuilt in brick and stone. Different perspectives on history start to pop up when you travel to your neighborhood countries.

St. Stephen’s Basilica keeps up with the majestic atmosphere of the city. Go up if you want to see the city from the top, because it is one of the two tallest buildings in Budapest and no other building is allowed to overcome its height of 96 meters.

Even though I am not particularly interested in those landmarks and have very little knowledge about their architectural or historical importance, I like the atmosphere that they create and the fact that in 2 days I have walked 47,000 steps, around 34 km and climbed 90 floors to see just a tiny portion of the city.

Parapark

Now about the cool stuff. Parapark is a game where you descent with your friends into a basement to crack some code, solve puzzles and reveal the mysteries. It is the classical doors puzzle game with real locks, keys, basement heavy smell, creepy background sounds and Xubuntu installed on an in-game PC. Go to their website and book a last minute game.

We had a team of 3 and it was perfect. While having the most incredible team, we still didn’t solve the mystery in 60 minutes, but we were close. Nevertheless our guide was kind enough to allow us to finish the game. My hands were dirty from all the dust around, we were sweating running around trying to fit a code to a safe, I was struggling to understand the walkie-talkie accent and it was awesome!

Cat Cafe Budapest

Few cities in Europe have cat cafes and Budapest is one of them. Cat Cafe Budapest is an adorable place with about 8-10 cats lying around, waiting to be pet by you, curiously inspecting the insides of your bag. There is a big lion-like cat, a newbie in the house, a grey elegant cat, long-haired pitch-black cat – for all tastes. The owner of the place is a very nice man, who sincerely loves all of his cats and high-fives with them.

Musical fountain

Margaret island, where Sziget Festival takes place, has a fun fountain. Go there in the evening at 9PM to enjoy music and lights. It gave me goose bumps listening to Verdi accompanied by the spectacular water show. It is not a world wonder, but it is a nice place to meet the sunset.

Small pleasures

They make your trip memorable. Like a gigantic ice-cream that you can buy at Oktogon intersection, old metro line of Budapest that resembles a way to hell, fun Erasmus people riding a beer bike, top of a fake boat and sinking Greek arc by the National Theater, big number of various fountains, not being able to find a specialty Hungarian dish that wouldn’t be also serve in Czech Republic.

Not so pretty

When you are visiting a place, you don’t want to see the ugly part of the city. But it stoke me how dirty can Budapest be. I suppose it was some special “throw-away” weekend, since by the houses on the streets there were mounds of garbage, like old cupboards, chairs, paintings, computer screens and other apartment waste.

Homeless people is probably also one of the city’s problems.

Locations

I have collected all the Budapest locations I have been to on a Google Map. Who knows, maybe you are going there this weekend and will find it useful.

View Budapest in a larger map

Packing light is hard

2014-06-26T00:00:00+00:00

Especially for girls. I used to pack up a giant travel luggage, even if I was going to be out for 2 days. “I dress according to the moods I sense in the room. As such, I must have options.” And my options were huge, same as my entire closet.

Times change and here is my first attempt to pack light for a 3-day trip. Start small with short trips and develop your personal mini-perfect-kit for longer journeys – that’s my motto.

Checklist

Without even considering other options, I have bought a backpack. Handbags are a lot more uncomfortable to carry. Suitcases? That isn’t going to be light. I have a Deuter GoGo with a volume of 25 litre in a girlish magenta arrowcheck color. Going beyond 30 litre already starts feeling big and not casual at all. The dimensions are also relevant to the airplane company that will allow you to take the bag on-board as hand luggage. The GoGo model is designed for city and traveling and lies within reasonable price range. I have purchased mine for 40 EURO. The killer feature of Deuter backpacks are the airstripes, which make your trips comfy, non-sweaty-back and hence enjoyable.

I usually have blind faith that I will be able to get some work done, therefore I always carry my laptop. It’s 14 inch display laptop weighing 1.31kg + the charger, whose weight is never specified by the manufacturer (they really should start doing that). If you don’t do constant work on your laptop and you use it to check facebook or skype, than a phone will do much better. A smartphone today is a super space saver. In the best case scenario it replaces the following accessories:

camera
laptop or tablet
books, newspapers, magazines, guidebooks
physical map
notebook
music player

Moreover it has a GPS and quite often represents an entertainment device. Oh, I forgot, you can make calls also. Along with the smartphone, I usually take a pair of earphones and my cable charger. Actually this time I will skip the charger and the power adapter, since I am visiting a friend and every house has a collection of micro USB cables. The point is to get maximum usefulness from multipurpose devices.

During a 3-day trip I don’t think I will get much reading done, so I am leaving my Kindle at home. At the same time I am not totally detached from my collection of books, since Kindle app syncs with your Amazon account and viva le cloud – I can read from my phone. Less convenient, but still better.

Next important step is the choice of shoes. It was one the most scrutinizing background jobs that lasted for 3 days in my mind. After solid market analysis, I have come up to the conclusion called Ecco Sport Biom Lite 1.2 Marry Jane. They look as if my grandma bought herself a pair of shoes to go to a health resort. Grandma will always choose cozy shoes and so they are. Being extremely light they offer a perfect foot support. Don’t go for converse or skate shoes, after 5-7 hours of walking you will see the difference.

I can wear them with trousers, shorts, skirts or dresses. They look equally okaish. The only annoying thing about Ecco Biom Marry Jane is that they are made of nubuck, which makes it hard to remove the stains. In one month or so I have made them look shabby and not so pretty anymore. Anyway totally worth the money.

Today I had the risk to buy the “neck-pouch-that-says-I-have-my-most-important-belongings-right-here-steal-it-asap”, in other words Samsonite Kangaroo Neck Pouch. I put in it my passport, travel tickets, credit cards, cash, coins. Most of my clothes lack any pockets, so I think it will become also a phone holder. Real reviews coming up after the upcoming trip.

Everything else sums up to this:

two t-shirts
pair of shorts
pair of flip-flops
swimming suit (I know I will be at a swimming pool)
bag with ladies stuff, tooth brush, razor and medicine (take only unusual kind, don’t carry any Strepsils with you – you can buy those in any drugstore)
3 pairs of panties
5 pairs of low socks
pair of skin color tights – wear them with the shorts in cold weather
comb – I have long hair
anti-perspirant – it is ginormous, definitely should find something more miniature
solution for cleaning contact lenses in a green bottle
mirror, marker for unexpected hitchhiking, labello lipstick, lighter, rolling paper and other small stuff

The little bottles of 100ml are simple, yet brilliant. You can buy a 4-pack from IKEA. I use them for shampoo, soap, solution. And the most awesome part is that you can take them on-board of an airplane.

Thank goodness it’s summer and my backpacks weights only 4 kg.

406 Not Acceptable items

Along with the checklist, there is a list of stuff I don’t take with me in the trip. I am not talking about a basketball ball or a frying pan, but about the things that I used to take. And so those are:

makeup – since I don’t use any, easy to do, but I know some girls might find it difficult living without at least basic makeup
hair products – I didn’t have the shampoo in my checklist, because I know I will be borrowing it from my host
books – occupy a lot of space and cause back pain
camera – if you are not a photo expert, smartphone camera probably will do
manicure set – 3 days, remember?

To conclude, captain obvious is ready for her trip. Wish me good luck and intriguing adventures!

Why travel?

2014-06-23T00:00:00+00:00

And Siobhan says people go on holidays to see new things and relax, but it wouldn’t make me relaxed and you can see new things by looking at earth under a microscope or drawing the shape of the solid made when 3 circular rods of equal thickness intersect at right angles. And I think that there are so many things just in one house that it would take years to think about all of them properly. And also, a thing is interesting because of thinking about it and not because of being new.

Now I feel abashed to say I travel to see and discover new things. If I insist on being honest with myself, than it becomes clear I lost any interest in the surroundings long ago. The reason might be the likelihood of old European cities – somehow they all look the same.

People (including myself) travel to show off in front of friends and family. Everybody enjoys sharing the beautiful moments of their life. You should have something to put on Instagram and brag about it. Otherwise what’s the whole point of visiting Tour Eiffel if you’re the only one knowing about it? I don’t necessarily see anything wrong with that – people, social animals, are predisposed to share with their peers and expect their positive acknowledge.

Traveling is stupid and a waste of time. Why then do I keep planing my next trip?

I love pinpointing my TripAdvisor map with new visited places and I feel good about myself. But this isn’t the main reason. Traveling drives challenge.

Everybody chooses different challenges. Some prefer extreme spots, others do stand-up comedy. By definition I consider myself an non-shy introvert with a mediocre creative impulse. Even being brave enough, it is still troublesome for me to ask for directions from unknown people on the streets, to maintain a lovely chit-chat with newly met people and to not look at them suspiciously. I remember that one time when I was crusading in circles around an area, because I was too afraid to ask for directions. Obviously that day I was one hour late for my first German language course.

When I go on a trip, I have this fear of getting lost, not being able to make my way to the final destination, being despised by people around, lose my flight, etc. Somehow during journeys I push myself to become brave, confident and powerful. New circumstances trigger my survival mode.

My favorite part of traveling is the transit from source to destination, being done particularly alone. I have these stories when a handle from my heavy bag rips off and I have to drag it for a long time, when I address the fast passing crowd to help me buy a ticket because the ticket-lady doesn’t speak anything else besides Polish, when I arrive at 5AM in Kiev and have less than 10 UAH to get from the bus station to the airport in time, when a Greek speaking Romanian allows me to wait inside the train wagon and calls my friends to pick me up, because it’s 2AM, cold and dangerous outside.

I pull myself together and do something I would never do. And since in my native city I know how to get around, I don’t even try to get out of my comfort zone.

Maybe it doesn’t make any sense to you, but it does for me. There is something about traveling, even now in the era of Internet, Google Earth and Google Street View. I choose to shape my life and personality with the power of inner and outer discovery.

I travel to challenge myself. Why do you travel?

Autoversioning static assets in Flask

2014-02-01T00:00:00+00:00

Whether you are actively developing a web application or just occasionally making small changes, you will most probably face the problem with browsers caching static content you have (JavaScript, CSS, images). Basically it means that your users will see outdated version of your app and won’t be aware of the brand new feature you were developing so passionately for the last week/month/year.

That’s why versioning exists and you will see how easy it can be done in an automatic way.

Requisites

The solution to caching problem can be implemented using any programming language, tool, framework, etc. Just to show you an example, I picked up Flask, which comes bundled with Jinja2 templating language for Python.

How it works?

Let’s say you have a CSS file and you included it in your HTML file. Now the trick is to give a different name to the CSS file when you are linking it to the template. But it must really tedious to change both the name of the stylesheet on hard disk and in the HTML. Therefore by using query string while linking in the HTML, we force the browser to reload the file and forget about the cache. At the same time there is no need to change the name of the file for real.

The version must be a unique number. Consider any of those:

file last modified timestamp
hash of the last commit
hash of the file

How to code it?

Most of the templating languages have filters. For example, in Flask and Jinja2 it is very easy to create a custom filer. Suppose you have configured Jinja2 to load templates and it works fine. Now it’s the time to create the custom filter.

import os
from some_app import app


@app.template_filter('autoversion')
def autoversion_filter(filename):
  # determining fullpath might be project specific
  fullpath = os.path.join('some_app/', filename[1:])
  try:
      timestamp = str(os.path.getmtime(fullpath))
  except OSError:
      return filename
  newfilename = "{0}?v={1}".format(filename, timestamp)
  return newfilename

<script src="{{ url_for('static', filename='js/script.js')|autoversion }}"></script>

So we use a decorator to create a custom filter. The usage is exactly the same as for any other standard filter, by using a pipe. The first and only parameter that our function accepts is the path to the static file.

Some manipulations are required in order to get the relative path to the file. For instance, we strip the first slash, otherwise os.path.join() will consider it an absolute path and will return this path. Then we need to get the timestamp of the last modification of the file, which is going to be a floating-point number that represents the number of seconds since the epoch. If you want you can surround it in a try-except clause in case the file doesn’t exist. The later is easy - just preppend the timestamp to the filename and return it.

Conclusion

A filter gave us an elegant and simple solution. Use the filter for any static files, whose contents will be updated sooner or later, but the name will stay the same.

If you want to see the filter implemented into a real project, then you can take a look at this blog’s github repo.

Update

Thanks to my friends, I have found out that query string to static assets is not generally a good option.

“Don’t include a query string in the URL for static resources.” It says that most proxies will not cache static files with query parameters. Consequently that will increase the bandwidth, since all the resources will be downloaded on each request.

“To enable proxy caching for these resources, remove query strings from references to static resources, and instead encode the parameters into the file names themselves.” But this implies a slightly different implementation :)

Erasmus for Moldova

2013-09-24T00:00:00+00:00

I will not tell you what is Erasmus Mundus in official terms, history background or program conditions. What I want to share is opportunities that exist for Moldovan students to enter an exchange program in one of the EU countries.

This article might be interesting for Undergraduate, Master, Doctorate, Post-doctorate and Staff levels. Though I can give more insight for Bachelor and Masters students.

Republic of Moldova is not part of the EU, so it is participating in a special project within Erasmus Mundus, called EWENT. And now starts the interesting part…

So, what is EWENT?

Seriously, do you want me to copy-paste from their official website? Generally most of the information can be found there, read it carefully. The information is in English, though there is also some in Russian. FYI if you have difficulties understanding the English version, you need to think twice before enrolling to exchange programs, which are held in English obviously.

On the Courses page you have a list of faculties available from EU partner universities. Don’t rush into decisions, if you didn’t find your faculty. Many courses are shared among faculties. For example, if you study Industrial Design, you might find a lot of courses from Construction Engineering, Civil Engineering, Product Design. I can tell from my experience that even thought I picked up Information Technology, half of my courses are not from FIT (Faculty of Information Technology). That’s somehow frustrating that Faculty of Mechanical Engineering has Artificial Intelligence and Neural Networks, Python for Computational Sciences, Java, C# programming and many other IT related courses.

Also it is a good idea to contact the local coordinator from the university you are interested in, if you have questions.

Take a look at the scholarships and what other costs the program covers (insurance, plane tickets). And if it says 1000€, then you will get monthly this exact amount of money. Usually for normal Erasmus, students get paid much less, so you can consider yourself lucky.

Who is eligible?

You must be a student or an employee to a high education institution (like a university) and not necessarily from UTM. Of course you need to have a Moldovan citizenship and must have not resided nor carried out some main activity (study, work) for more than a total of 12 months over the last five years in one of the European countries. I thought this last statement might be important for those who already have been to some exchange programs. Also you can conclude that a double citizenship is not a problem, as long as your “main” country is Moldova.

Don’t hesitate even if you are in your last year, it’s totally fine to come to Erasmus and write your thesis here (actually will experience that in my next semester :D).

Motivation

That might be not exactly obvious to everybody, why go to Erasmus at all. So I say Erasmus is awesome, because:

it’s a fresh look at studies

It’s a new university, new professors, different courses, classes, cafeteria, dorms.
it’s a new country to get to know

I am pretty sure that during a semester or two you will know where best food is served or where do I borrow a bike for the weekend.
you travel around the world (or EU at least)

Everything is much closer and accessible than it is from Moldova.
you make new friends

There are 500 international student here at CVUT in Prague.
you are cared about

Local coordinator, students, student organizations will guide you through the most important steps to provide some accommodation, necessary documents, transport pass, etc.

Also any of those points can be extended with lots of cool moments that are awaiting for you.

How can I apply?

The official application is on EWENT. Mind the deadlines and watch for the opening of the registration. As for the Erasmus period 2013/2014 the application was open until the start of January 2013.

Until then start preparing all the documents and contact our local coordinator Mariana Vrîncean (her email is on the website too). She can prepare some university documents for you and give you feedback on your application. Keep in touch with Mariana, because she can provide a lot of useful information. Moreover she participates in the decision making process, so it is good if she got to know you.

What can I do to be selected?

For short:

meet the application deadline
be sure all the documents are uploaded and in order
pay attention to the Motivational Letter, talk about our interests (in studies, research, development), list your strong parts, mention what would you like to do during Erasmus and what impact will it have on your future career
have a decent CV. It totally depends on your personal achievements, but usually it is important to prove that you are a proactive persona.
your marks - no, you shouldn’t be a genius from the top 5 university students. I even doubt that grades have any serious weight.

In reality there’s is no general recipe of how students get selected. We can only guess what is appreciated more and what is skipped from the grading.

I am an Erasmus student

Happy news! You were selected.

The organizers of EWENT from Poland will be responsible for your Learning Agreement, which you need to sign. Also after you will get your visa, they will send you your Erasmus insurance and round-trip plane tickets.

From now on much of the activity depends on the hosting university. They must send you necessary documents to get a visa. They can ask you to pick up courses. There might be lots of other details. In my case I was contacted also by the student organization ISC. They kindly provided a contact person that picks you up at the airport and helps you with accommodation.

P.S. the website layout or some details might be outdated to the moment you are reading, so be sure to make yourself familiar with the latest news from EWENT.

OMG! git reset --hard

2013-08-28T00:00:00+00:00

I bet you know that jaw-dropping command git reset --hard <commit-hash>. Remember that first time you lost all that code? No? Well, good for you you didn’t.

Fair enough the command is considered potentially dangerous. I have even seen blogposts about git reseting that make it hard to detect the command in-need (even if the user came for a full delete/erase/destroy this commit and the data it contains). The authors add lots of explanations around, so you need to read first and then type the correct command. No copy-paste, no tl;dr allowed.

In fact as I learned from Pro Git the dreadful harm is reversible…

Lost my commits

So let’s take an example. Suppose you just have started a new project and your history looks like this:

$ git ls
f1c16f4 (HEAD, master) second [ana-balica]
743c90d first [ana-balica]

* ls is an alias for some pretty formatting

Now you type git reset --hard 743c90d, which resets your HEAD at 743c90d commit. But actually that last commit is not lost. For a recovery all you need to know is the SHA of that lost commit. Create a new branch starting with this commit, rebase or merge your master branch with the new branch that contains the commit and you are saved.

$ git branch recover-branch f1c16f4
$ git rebase recover-branch

It’s good if you have the SHA of the lost commit(s). If not, try those steps in the following order:

check the terminal output, at some point you have typed git log and maybe you have the hash
try git reflog - it records your branch tip changes. Look for the commit name if you remember it. Mine looks like that:

$ git reflog
743c90d HEAD@{4}: reset: moving to 743c90d
f1c16f4 HEAD@{5}: commit: second
743c90d HEAD@{6}: commit (initial): first

use git log -g if you still can’t comprehend which is the one

Lost stagging area

Worse then reseting to a commit in the past, it is reseting the HEAD - git reset --hard HEAD. But don’t worry if those were in your staging area, which means your index knew about them. Type git fsck --lost-found. It checks the validity of objects in the database and therefore you will see your dangling blob there. You can open it and check it’s contents. For recovering all that data simply dump it’s contents to the file git show 898910b > test.txt, where the SHA is the blob’s SHA.

Lost unstagged area

The horror is here! At this point you have made some changes to your files, never git added them, and then the magic happen and wiped away all that. Must be a crisis in your life, you are doomed, life doesn’t make sense any more…

Maybe you are lucky and you can find the light at the end of tunnel. Check that:

maybe Ctrl+Z?
see if your smart IDE has any internal version control system, personally I use PyCharm and it is very easy to view and recover from history. Protect yourself, use an IDE ;)
are the changes recorded anywhere else? For instance if your project is located in Dropbox (I know that’s stupid if your have version control, but who knows), then you can see previous versions on Dropbox website and restore to any version you want. A new version is recorded on a new save. Don’t leave it for tomorrow, because only the last 30-days versions are stored in the cloud.

If you have some other ideas on code recovery, feel free to share :)

SiCamp Moldova 2013 speculations

2013-07-29T00:00:00+00:00

Social Innovation Camps are ok. Moldova also has one, I wonder why it’s not marked on the map of Global SiCamps and no words about it even here. Seemingly we are apart from the rest of the world as usual.

Get it working

This year teams were encouraged to get some real working done. I suppose that includes launching the application live. As for me it’s so hard to deploy it when you know the app is half done. But that’s sometimes the right thing to do, because otherwise you will be having lots of those “nice” apps in your repositories, that will rust there till the end of the world. So just launch it.

Nevertheless I hate hurry projects, maybe it’s better to fail fast and get a prototype done. Reasons:

the working app will keep reminding about itself so you don’t abandon it
users will tell you what’s best, so you get some real feedback
you already started solving that noble social problem - get yourself a beer

Are the SiCamp projects really going to hold on? We will see.

Mentors

Mentors on such events are a pain in the ass. That’s what they do:

giving lots of ideas - it’s 48 hours and about 20-25 hours of real productive work and it’s hard to stay focused when every single mentor has an overwhelming conglomeration of ideas in his head. The ideas might be very useful, but usually each person’s perspective is killing a strong fixed aim of the team. The best way is to give mentors a single person from the team to fuck with his ears.
giving great ideas about the stuff they are not experts in - i assume this happens because of the status they got. Maybe they are trying their best to be useful, but sometimes it’s just not working for anybody.

In the end

I have no idea if the event is worth going to or continued to be organized. When you look at the big picture, it absolutely plays no role. People that will be promoted to the status of “they changed the world”, will probably do it anyway without SiCamps. Thinking less globally, one can see lots of benefits in creating and participating to such events.

Custom Python Markdown Extension

2013-07-16T00:00:00+00:00

Markdown is awesome. It’s a pleasure to use it and it’s a pleasure to extend the Python Markdown module to fit your needs. Along with a bunch of official extensions, the module contains a nice API for writing your own extensions.

The API is clean and comprehensible. The reason of writing this article is that maybe somebody else will bump into the same problems like me and might find some of the solutions good enough to be used.

Problem statement

Right now I am writing in Markdown. When I want to insert an image I am doing the following ![My Image](/static/img/my_image.png). I specify the relative path, because when I switch to different domain names, I want to have the same picture and no broken links. So I figured it out there can be 2 ways to solve that:

render the template before markdown does. Use your templating language for inserting a variable that will be your MEDIA_URL.
write a markdown extension that will fix your links

At this moment I would choose the first solution. But I implemented the second one, so I want to share some basic knowledge.

My setup

I use Flask-FlatPages for rendering articles. Internally Python markdown module is used, and that means that you build your own extension easily.

Consists of

The extension must have:

a class that will represent the extension and needs to inherit from markdown.extensions.Extension and implement extendMarkdown method
a function called makeExtension(), since you will be having per module one single extension. When you will specify the name of the extension (i.e. to the configuration variables of Flask FlatPages or directly when creating the Markdown instance), it will import the module and call that function. Here I am just quoting the documentation.
some kind of processor that will perform the manipulations with your markdown text

Let’s start coding

def makeExtension(configs=[]):
    """ Return an instance of the AbsoluteImagesExtension """
    return AbsoluteImagesExtension(configs=configs)

This function also accepts a list of configs. It is useful for us, since we want to get a base URL or a media URL which we will prepend to the image source links.

There are different flavors of processors: preprocessor (before code is sent to Markdown core), treeprocessor (being an ElementTree object), postprocessor (you got your output string already). For my needs I have decided that an ElementTree is a good way to easily extract <img> tags and insert the changed src.

class AbsoluteImagesTreeprocessor(Treeprocessor):
    """ Absolute Images Treeprocessor """
    def run(self, root):
        imgs = root.getiterator("img")
        for image in imgs:
            if self.is_relative(image.attrib["src"]):
                image.set("src", self.make_external(image.attrib["src"]))

    def make_external(self, path):
        return urljoin(self.config["base_url"], path)

    def is_relative(self, link):
        if link.startswith('http://'):
            return False
        return True

Method run() is mandatory. The syntax is straight forward and the implementation is pretty naive. Also pay attention that we have the access to the config.

And finally the extension class:

class AbsoluteImagesExtension(Extension):
    """ Absolute Images Extension """

    def __init__(self, configs=[]):
        self.config = {
            'base_url': [None,
                         "The base URL to which the relative paths will be appended"],
        }

        for key, value in configs:
            self.setConfig(key, value)

    def extendMarkdown(self, md, md_globals):
        absolute_images = AbsoluteImagesTreeprocessor(md)
        absolute_images.config = self.getConfigs()
        md.treeprocessors.add("absoluteimages", absolute_images, "_end")

        md.registerExtension(self)

self.config should be a dictionary and store values of the form param_name: [param_value, description]. In the extendMarkdown() we are adding our treeprocessor class to Markdown TreeProcessors by specifying it’s name, an instance and where to insert it - before the end.

The extension lives here - https://gist.github.com/ana-balica/5944798.

Enabling the extension

For Flask do something like that - FLATPAGES_MARKDOWN_EXTENSIONS = ['absolute_images(base_url=http://this-important-url)'] in the config. First Markdown will search for the extension in the module directory - markdown.extensions. If you don’t want to keep it there for some reasons, you can make the extension name contain dots (then Markdown will import it as-is) or prepend to the extension filename mdx- (mdx-absolute_images). But be careful that the mdx- notation is discouraged from being used.

Welcome to Design Patterns

2013-07-03T00:00:00+00:00

There are many recommended books and articles to be read on Design Patterns. High variety leads to a problem of choice. So based on the magic of rational randomization I decided to read and understand Head First Design Patterns. I am a Head First virgin, so I even read the pre-introduction with explanations on metacognition, pretty pictures and redundancy. So far, so good.

The reason of writing all this down is to overflow my brain with redundancy, “do my homework” by summarizing everything read in a chapter and maybe leave a note for future me and all other Internet readers about something important. Moreover that’s going to be a shortcut on the book (though the book is funny).

One chapter → one article. And the first one is called Welcome to Design Patters.

Encapsulation

The first announced Design Principle in the book is:

Identify the aspects of your application that vary and separate them from what stays the same.

That is the guru of all Design Patterns. That means that all patterns are trying to accomplish this in some way or another. It is very important to separate what varies from what stays the same, since changes are too frequent in IT world.

The examples in the book are based on ducks. We will examine a very similar situation based on the Design Puzzle. Initially we have a very simple application that is made of a simple character hierarchy.

That’s a very inflexible hierarchy and will lead to many problems over time:

What if Queen doesn’t fight? Should you overwrite the method in this class?
Are there going to be more characters that don’t fight? Will you overwrite the behaviour in every class?
Trolls fight different from Kings!

Ok, that’s the moment when you should SHIFT+Delete your code. Kidding :) You need to apply that one first principle and conceptually do the following:

Interface

That green spot represents all the Fight/Weapon Behaviours. It’s not a good idea to create just a bunch of classes, since you will lose the ability to use polymorphic behaviour and will hard-code each behaviour to each character. Create an interface. Side-note: here the interface doesn’t mean necessarily a Java interface - it’s a conceptual interface, a special class that holds the public declarations and has no proper implementation (also not always true, think about abstract classes in C++ which can contain implementation).

Program to an interface, not to an implementation.

Here’s what we have done after learning that interfaces give you lots of flexibility. In your Character class you will create a property of type WeaponBehaviour. Here, let me show you the code.

public abstract class Character {
	WeaponBehaviour weaponBehaviour;

	public void setWeaponBehaviour(WeaponBehaviour w) {
		this.weaponBehaviour = w;
	}

	public void fight() {
		weaponBehaviour.useWeapon();
	}
}

public interface WeaponBehaviour {
	public void useWeapon();
}

public class KnifeBehaviour implements WeaponBehaviour {
	public void useWeapon() {
		// an implementation of a fight with a knife
	}
}

public class Queen extends Character {
	public Queen() {
		this.weaponBehaviour = new KnifeBehaviour();
	}
}

Moreover by having our WeaponBehaviour setter, we can successfully teach our Characters to learn how to use new weapons at runtime.

Composition

A perfect inheritance is a IS-A relationship. It is actually more than just a IS-A according to L principle from SOLID. Nevertheless what we have done here is eliminating IS-A and using HAS-A, which is composition.

Favor composition over inheritance.

It gave us all those benefits of behaviour change at runtime and the encapsulation of a set of algorithms. Now not only Character descendants will be able to fight and use weapons, but also any other class can be taught by the power of composition.

By the way, all that talk about fighting characters is a patters called Strategy.

The Strategy Pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Strategy lets the algorithm vary independently from client that use it.

Strategy is a behavioural pattern. If you want to learn more, check this out.

Conclusion

xkcd getter

2013-04-25T00:00:00+00:00

Some time ago I had an assignment to implement a crawler. Now I understand that I didn’t do the right thing, since a crawler is a program that will follow links in a systematical manner on every page. What I did is called a spider - a program that downloads web pages. Well, you can take your spider, add some parsing, extract the necessary links, follow them, download web pages and basically this is a crawler.

Nonetheless I still have that spider written in Python. It suffered many changes, since initially it was implemented using sockets, so far now it uses the urllib. Since I enjoy reading xkcd, this website became my target.

The idea to create this post came out of this comic:

Tried my spider on it and it didn’t crash, gave me the title, the caption and the picture - yey!

Text or screencast

If you are a visual person, then you can watch the screencast below. It describes every single line of code written. Important: the version of the script used in the screencast is a bit outdated, some design solutions were changed, but if you are interested mainly in parsing, then the video is eligible to be watched.

If you don’t like watching looong screencasts (like me), then here is a short overview of the script. Also you can find the source code right here.

Prerequisites

You will need:

Python 2.7
lxml library
optionally a browser extension to check you xpaths. I am using XPath Helper for Chrome

Downloader

Create a separate class that will hold the download functionality. Here is the snippet:

class Downloader():
  '''
  Class to retreive HTML code
  and binary files from a
  specific website
  '''

  def __init__(self, url):
    self.url = url

  def download(self, image_name='', is_image=False):
    try:
      browser = urllib.urlopen(self.url)
      response = browser.getcode()
    except:
      print 'Bad connection'
      sys.exit()

    if response == 200:
      contents = browser.read()
    else:
      print 'Bad header response'
      sys.exit()

    if is_image:
      self.save_image(contents, image_name)

    return contents

  def save_image(self, contents, image_name):
    image_file = open(image_name, 'wb')
    image_file.write(contents)
    image_file.close()

Pass to the constructor the URL where from to download the contents. There are 2 methods, from which only the download() represents the interface of the class. urllib.urlopen() method opens a socket for you establishing a connection. It is a good idea to wrap this part of code into a try/except since it can raise an exception if the connection cannot be made (many reasons for that). Afterwards check if the server responded with 200 OK and in case of success get the contents. Those contents is the HTML code of the page.

There is also a scenario when the downloaded content is an image (binary file). For that we need explicitly to pass the is_image=True parameter. An image will be saved on the local machine in the current directory (where the script is placed).

Parser

Parser class is a not a generic class. It is build specifically for parsing xkcd.com. If you plan on extending the spider, then it might be a good idea to create an interface Parser and let other classes implement that interface.

Since the xkcdParser class is a bit long, we will examine it in chunks.

class xkcdParser():
  '''
  Class for parsing xkcd.com
  '''

  def __init__(self):
    self.url = "http://xkcd.com/"
    self.last_comic_nr = None
    self.contents = ''
    self.title = ''
    self.caption = ''

  def set_last_comic_nr(self):
    downloader = Downloader(self.url)
    self.contents = downloader.download()
    self.last_comic_nr = re.search(r"http://xkcd.com/(\d+)", self.contents).group(1)
    self.last_comic_nr = int(self.last_comic_nr)

  def get_title(self):
    if self.contents:
      tree = etree.HTML(self.contents)
      self.title = tree.xpath("string(//div[@id='ctitle'])")

  def get_caption(self):
    if self.contents:
      tree = etree.HTML(self.contents)
      self.caption = tree.xpath("string(//div[@id='comic']/img/@title)")

  def get_comic(self):
    if self.contents:
      tree = etree.HTML(self.contents)
      url = tree.xpath("string(//div[@id='comic']/img/@src)")

      downloader = Downloader(url)
      downloader.download(self.title, True)

Keep the URL of the website in a class attribute. Besides the constructor there are a bunch of methods here.

set_last_comic_nr is… well self-explanatory. Why do we need it? Because we plan on allowing the user to specify the number of the comic and also generate a random number in a specific range. Not the best solution to create a Downloader object inside the method, but it works for this short spider.

After you get the contents, you need to parse them. Two solutions (might be more, but the most mainstream) are xpaths or regular expressions. If you go to the home page of xkcd.com then below the image you will see a permanent link for this comic and it has the id we are looking for - the id of the most recent comic. Using the inspector we can check that this part of HTML is not wrapped into a tag, the text is placed between  s after an ul. That’s definitely not a well-formed XML mark-up.

Even in this case it is possible to get the data with the following xpath - //div[@id='middleContainer']/text()[preceding-sibling::br][1]. That will return the following string - ‘Permanent link to this comic: http://xkcd.com/1203/’, which again should be parsed with regular expressions.

I’ve taken a shortcut. My regexp searches for one or more digits after the string http://xkcd.com/ through the whole DOM and groups those digits. Regular expressions are slow, in my case it was necessary to build a whole automata out of the contents, therefore you need to avoid them as often as possible.

The other 3 methods are already using xpaths to search for the title, caption and path to the image through the contents of the page. The method get_comic() also instantiates a Downloader object and downloads the image.

The xpaths are fairly simple, but anyway let’s examine of them. How about the string(//div[@id='comic']/img/@src)? English translation is the following -

search through the DOM for a div with an id that equals ‘comic’
inside that node search for an img tag
get the src attribute of that img tag
strigify everything you have found - if you don’t use the string function, then xpath will return an array with every matched element.

The rest of the code looks like that:

class xkcdParser():

  def get_current_comic(self):
    self.set_last_comic_nr()
    self.get_title()
    self.get_caption()
    self.get_comic()

  def get_comic_by_id(self, comic_nr):
    if not self.last_comic_nr:
      self.set_last_comic_nr()

    try:
      comic_nr = int(comic_nr)
    except:
      print 'The comic number should be an integer'
      sys.exit()

    if comic_nr <= self.last_comic_nr:
      url = self.url + str(comic_nr)
      downloader = Downloader(url)
      self.contents = downloader.download()
      self.get_title()
      self.get_caption()
      self.get_comic()

  def get_random_comic(self):
    if not self.last_comic_nr:
      self.set_last_comic_nr()

    comic_nr = random.randint(1, self.last_comic_nr)
    self.get_comic_by_id(comic_nr)

We need 3 more methods: get the latest comic, get a specific comic and get a random comic. get_current_comic() actually has a problem, because it hides the loading of contents. Since we call that method that will set last comic number, we know that it will also set the contents ad there is no need to download the page again. Still, prefer a different implementation for that.

Before downloading a comic with an id, first check the data that you get from the user - if it’s an integer and if it’s in range, perform the download, setting of title, caption etc. The get_random_comic() uses the method above, initially generating a random number.

Aaaand we are done!

You can get the entire snippet by accessing this gist - https://gist.github.com/ana-balica/5454270

CLI and GUI git

2013-04-21T00:00:00+00:00

To be clear from the very beginning, I vote for CLI git to start with. There’s nothing better than understanding what exactly you are doing and not just try and click every available button. Just imagine yourself connecting to a remote server that doesn’t have your favorite GUI client.

During my interaction with git (which started not so long ago), I’ve developed a hybrid workflow. There are lots of git clients and to be honest, I haven’t tried many of them, so I’ll just present the one I’m familiar with - gitg.

History

Visualizing your tree in git is definitely a very useful thing. To achieve the following you can use the --graph argument. If you leave the default log command then the long commit messages will make it hard to read the topology.

Use pretty formatting and create an alias for that:

[alias]
ls = log --pretty=format:"%C(yellow)%h%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate

Therefore the command git ls --graph will output the following:

\*   46be671\ Merge branch 'development' of github.com:author/repo into development\ [author]
|\
| * fe8bef7\ commit nr 5\ [author]
| *   b801dc6\ Merge branch 'development' of github.com:author/repo into development\ [author]
| |\
| | * ace6bdb\ commit nr 4\ [author]
| * | 2fd916a\ commit nr 3\ [author]
| * | 7419237\ commit nr 2\ [author]
* | | fd0c4bf\ commit nr 1\ [author]

Not bad, but a GUI client makes it better. Basically it’s the same branch topology , but it offers an intuitive way to crawl though commits, see the details, files that were changed, the branch pointers.

Diffs

git diff shows changes between working tree, commits, branches, etc. It can take lots of arguments and is very flexible. In case you just don’t want to remember all those possibilities, the client will provide snippets with diffs for lots of scenarios. It becomes handy when you want to review what you are about to commit or what what do you want to add to your staged area, what is inside your stashes, what have you committed long ago.

Stash

I use stashing all the time. And I also try not to keep them there for a long time. So mainly I do something like this:

$ git stash
$ do some stuff here
$ git stash pop

When you pop a stash, this means that you apply your stash to the current working tree and also drop it from the stash list. That’s a handy shortcut.

If you want to check the diffs inside your stash type git stash show -p stash@{0}. Not the most pretty stash name (while typing you need to reach for @ and curly braces), so I find it quite nice to visualize the list of stashes and the diffs in gitg. The annoying part is always choosing the branch where to apply the stash, there’s no ‘please make the current branch default for stashes’.

Blame

The client git is pretty straight-forward and has only the ability to blame an entire file. Hence it does git blame <filename>. Still useful, though personally I think would be nice to get a link to the commit from the blame section.

Patch

Very often I find myself willing to add to the staged area only several lines from a file. Then I do git add -p and add just the hunks that I want. With gitg I choose the Commit tab and the file I want to patch and simply by holding CTRL and clicking on the lines, I can choose exactly what I want to stage.

The client has this nice feature to stage specific lines, but at the same time it can be quite annoying when you got big chunks of code and you want to make it fast. Some git clients allow to select lines and stage them all at once, unfortunately not gitg.

Conclusion

At the beginning I though that GUI git is going to be presented as a very handy tool, but eventually I got convinced that it barely associates with the CLI power.