Meet Gor - Type: til

Tmux scroll mode select and copy

Wed, 30 Jul 2025 00:00:00 UTC

I have found scrolling in tmux is a bit unconventional. Maybe its just me, but sometimes, the terminal interfaces are different, sometimes they are a bit wired, I use Ghostty right now, so selection without tmux works a charm, but the moment I am in tmux, ahmm, it kind of breaks. The selection of text doesn't work with the mouse atleast, so I thought, I need a keyboard centric selection and copying mechanism, and dug in this rabit hole. Turns out, tmux is way better in selection then zellij, the multiplexer I had been using due to scroll and selection issues in terminal interfaces. It's simple as `Prefix + [` to enter the scroll mode, `Shift+V` or `Ctrl+space` or `Ctrl+v` to start the selection in different ways, and finally, `Enter` or `Ctrl+j` to copy to clipboard. There is a dedicated mode for vi users, to feel at home. Tmux + Vim again a deadly combo. ## Entering Scroll Mode To enter scroll mode, I had this config setup in my `.tmux.conf` file: ``` set -g mouse on ``` However if you want to enter without mouse on, you can use: ``` Ctrl + b + [ ``` Here the `Ctrl + b` is for the prefix key, and `[` is the key for enterring into the `Scroll Mode` From here on you can use `hjkl` or arrow keys to scroll up or down. To quit, you can press `q` or hit escape. ## Selecting Right now in scrolling mode, you can only navigate to the next or previous line. But you can use `Ctrl + ` to select any arbitrary sections of the terminal output. There are two modes in tmux for this the `copy-mode` i.e. the default, and the `copy-mode-vi`. Without any config, this is be set to the default `copy-mode`, so we can use `Ctrl + Space` to begin selection in the scroll mode. To get more keybindings for the copy-mode and copy-mode-vi, use For copy-mode ``` tmux list-keys -T copy-mode ``` For copy-mode-vi ``` tmux list-keys -T copy-mode-vi ``` This is a default and can be over-written by using the following config: ``` setw -g mode-keys vi unbind-key -T copy-mode-vi Space # replace with the key combination you want bind-key -T copy-mode-vi send-keys -X begin-selection # Example: using ctrl + b + v as the key to start selection # bind-key -T copy-mode-vi v send-keys -X begin-selection ``` This all would be applied to copy-mode if you are not changing the keybindings for copy-mode-vi. ## Copying Once, you have the selected region or piece of text, you can copy that to the clipboard (that is what you do, while logs-driven debugging :). By default, you can use: For copy-mode - `Ctrl + w` or For copy-mode-vi - `Ctrl + j` or `Enter` to copy the selected text to the clipboard. This is default, and again can be over-written by using the following config: ``` setw -g mode-keys vi bind-key -T copy-mode-vi send-keys -X copy-pipe-and-cancel # Example: Remap 'y' in copy-mode-vi to copy selection and exit # bind-key -T copy-mode-vi y send-keys -X copy-pipe-and-cancel ``` This all would be applied to copy-mode if you are not changing the keybindings for copy-mode-vi. That's it this should be more than enough for most of the log-driven or printf debugging tasks.

Getting Python version in a UV project

Sun, 01 Jun 2025 00:00:00 UTC

I was live streaming, creating a AI Coding Agent from scratch in Python. I was excited to create the agent. You can watch the [livestream](https://youtube.com/clip/Ugkx2AcRqaYcMgAqZgtA2k48iYb8sJf9VL6F?si=XTtC2zFs2Kloia0L) here. Anyways, the thing is the uv python project has a `. python_version` file that will store the python version. I thought, it might be used from `pyproject.toml`, or other setup files. But that is really neat from uv.

Add hunks: only line specific changes with git add -p

Fri, 11 Apr 2025 00:00:00 UTC

I had made the changes for a future release, next set of changes so to speak in git terms. Then, I encountered a bug while testing the previous changes. Now, I could have 1. Open VS Code and add that little `+` icon in the gutter to fix the bug 2. Create a separate branch with stashed changes for this new feature 3. Use GitSigns in NeoVim 4. Use `git add -p` to stage [hunks](https://git-scm.com/book/en/v2/Git-Tools-Interactive-Staging) The last one was a gem given by ChatGPT, and then I went a bit rabbit hole for reading about it and it turns out it's real! It's called staging patches, and it has an interactive mode to select or reject each hunk (patch, line of changes) at a time. So by entering the command `git add -p` this will let you go through each file and individual change (patch/hunk) and take action interactively. You can use the below commands for moving one hunk or patch at a time: - `y` for adding (staging) the current hunk (patch) - `n` for not staging the current hunk Or you can even use the below commands for moving one file at once: - `a` for adding (staging) all the patches in the file (just like `git add file.txt`) - `d` for not staging the file at all, just skip the file for patching If there are multiple hunks (patches) in the file, the prompt will show the other options like `[y,n,q,a,d,j,J,g,/,e,?]` ``` (1/7) Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? g 1: -33,6 +33,7 +from app.utils.utils import get_additional_param 2: -345,6 +346,7 + strict_check = str_to_bool(get_additional_param("stri .... .... 7: -875,7 +878,7 - if strict_check: ``` This shows the `g`, `j`, `J`, `/`, and `e` which are quite helpful for navigating across multiple patches (hunks) in the file. I haven't explored all of them, but the 4, I showed earlier, are the ones that I think I will be using extensively now, within the terminal. No need of fancy editor shenanigans and redundant branching tricks. Git Good!

Add hunks: only line specific changes with git add -p

Fri, 11 Apr 2025 00:00:00 UTC

Format JSON in Vim with JQ

Sat, 05 Apr 2025 00:00:00 UTC

We can use

:%!jq .

To instantly format a json file opened in vim/neovim. I relied on online foramtter and opening vscode for formatting. But this quick tool just saves a lot of hassle. Opening VS Co** == shutting down the laptop, it clogs the memory badly, so I avoid opening VS Co** as much as possible.

Yes, to use this, you need to have jq installed on your system.

If you are elsewhere and want to format json file using jq, you can quickly do this:

jq . input.json | sponge input.json

# OR

jq . input.json > tmp.json && mv tmp.json input.json

If you are in Vim and want to format other file, just add ! at the start of the command and that will run in the shell for you, so you don't have to exit vim.

Zellij Open Scrollback Edit mode

Thu, 20 Mar 2025 00:00:00 UTC

I was looking at a long list of logs, (debugging of course). I had a list of transactions - Two list of transactions, one as ground truth and other as predicted. Transactions is a list of object(dictionary or map), each object has fields like date, amount, description, etc. - Those objects in a list need not necessarily be in order, however I do want to compare them, how to do that? - I decided to sort them based on date(one field in that object). Then for each date, I group the transactions and this would narrow down the search space for comparison of one-one transaction, since now I can compare which of the ones are closely matching, the date will be a exact match the amount should also be, the description can be fuzzily matched. However, for amount, I guess I was wrong, there could be a value like `10` and the other could have `10.01` and those python doesn't count equal atleast when compared as a string. I converted to float and compared rounded off numbers. Now the problem kicked in and print statements flooded, dates everywhere. Now I was using ghostty terminal, and there was definitely some scroll limit so I couldn't scroll all the way to start. So I opened up zellij. I had a log with *MATCHED* text where I logged the date and amount where both transaction matched. Now, I wanted a count of these matches. I could use search but it was not giving the count of occurrences, I can't keep counting with my mouth(that idea flew by though) Now, that's where I accidentally hit `S` and `E` And I was in a editor, woah! I was excited, I could finally copy and throw that in vim and get everything I want. hehe I tried +"y but it didn't copy to the clipboard, it yanked yes, but not in the system clipboard. That frustrated me and took my hope down, but I googled it and also gpted it (is that a word, I think we can say llmed it). And yes we can set the `export $EDITOR` to the editor executable path and it would open that thing in that editor. I did and it did work. That mode is called [scrollback-edit](https://zellij.dev/news/edit-scrollback-compact/) mode. I should say a life saver mode, a log viewer and really cool. I am probably too dumb and I know this exists in tmux, but I felt good and helped me solve my problem. So thank you whoever made that mode, its really helpful to debug with logs (debloging) Yes I am bad at naming things, but I like this more than vibe coding ;)

Sample TIL

Sat, 30 Nov 2024 00:00:00 UTC

## What I learned This is the content of the TIL ## Conclusion Happy Coding :)

Gotcha with Chained Assignment in Python

Wed, 27 Nov 2024 00:00:00 UTC

I was writing some Python code and wanted to initialize a few variables to an empty list. Instead of creating separate lists for each variable, I decided to use chained assignments like this: ```python a = b = c = [1,2,3] ``` It seems okay, nothing new, we are just assigning a, b, and c as an empty list. But notice the problem if you can. If we try to change the value of b, what do you think is going to happen? ```python b.append(4) ``` If you didn’t know about how Python handles assignments, you might expect the variables to behave like this: * `a` should still be `[1, 2, 3]` * `b` should be `[1, 2, 3, 4]` (since I modified b) * `c` should remain `[1, 2, 3]` ```python # You might think it will be this # a = [1,2,3] # b = [1,2,3,4] # c = [1,2,3] ``` But, hello python, it hides the pointer magic behind this statement The actual and expected state of the variables is: ```python # But it is actually this # a = [1,2,3,4] # b = [1,2,3,4] # c = [1,2,3,4] ``` When you chain the assignment like this, all three variables refer to the same list object in memory. This means, that all the variables will hold the same object, this means that if you change any of the variables assigned that reference it will change the object, and that will result in changing all the variables since they are referring to the same variable. You are not creating three independent lists. Instead, all three variables (a, b, and c) are referencing the same list object in memory. They don’t hold copies of the list, they all point to the same object. In Python, this is called **reference assignment**. When you modify one of the variables (like appending to b), you’re not modifying just b, you’re modifying the single list object that all three variables are referencing. Since all three variables point to the same list, any change you make to the list will be reflected in all three variables. On the other hand, if I do the same thing with strings, like this: ```python a = b = c = "hello" b = "world" # a = "hello" # b = "world" # c = "hello" ``` This only mutates the b variable and not the a and c, since string is not a mutable object in Python. In Python, the objects are either [mutable or immutable](https://realpython.com/python-mutable-vs-immutable-types/) Some of the primitive data types are: Mutable Types: * List * Dictionaries * Set * Byte Array Immutable Types: * Integer, Float, Complex * String * Tuple * Bytes * Boolean * Frozenset So, suppose you assign multiple variables with the same value of a mutable type. In that case, the change in one variable will mutate the other variables as well since the underlying object in memory is the same. So, this is what I learned from the mistake, avoid the chining assignment when dealing with mutable objects Instead do the following: ```python a, b, c = [], [], [] b.append(256) # a = [] # b = [256] # c = [] ``` This is safe and the right way to assign variables to individual values instead of the same value being referred to by all the variables. Happy Coding :)

Building Golang from Source v1.23 and Above

Tue, 19 Nov 2024 00:00:00 UTC

## Introduction Are you excited to try out the latest golang version, or test out your changes (cooking some serious stuff?), or install some random golang version? Then let’s explore one of the easiest ways to install golang on your system (Linux). ## Building Go from source 1.23+ The process for installing and building golang from source is quite simple. * Clone the repo * Build the binary * Set Environment Variables * Export the binary to the system path For detailed other steps, you can follow [this guide](https://go.dev/doc/install/source). ### Clone the repo Just clone the repo from GitHub or Google Git repo. ```bash git clone https://github.com/golang/go OR git clone https://go.googlesource.com/go ``` This will install the golang source code required to build the golang binary and the ecosystem (gofmt + standard library + test suite). Then let’s navigate to the cloned repo and we can build the golang from the source. ### Build it We need to run the bash script in the folder to build the binary. They `all.bash` can be run to build the binary which will be stored in the `go/bin` folder `go/bin/go` and `go/bin/gofmt` files. These two binaries will be generated and are required in the Golang ecosystem. ```bash cd src ./all.bash ``` Once we have the binaries in the specified folder, we can move ahead to make the environment understand where the actual binary is located. ### Environment Variables The `GOROOT` and `GOPATH` variables are required for the golang ecosystem to work as expected. The `GOROOT` is set as the path to the actual golang source repository, the cloned repository from which we built the binary. This `GOPATH` is the path where Golang stores the external repositories or modules for use anywhere in the system. ```bash export GOROOT=path_to_clone_repo export GOPATH=$(HOME)/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin ``` The `PATH` environment needs to be updated with the `GOROOT` and `GOPATH` to make the binaries in those paths visible and accessible to the system. > NOTE: If you are installing the golang from source when you have already a version of golang installed on your system, then you need to make sure you do not mess up the `GOROOT` and `GOPATH`. > > Those could be juse set with the current shell session, as you do not want this golang version permantely on the system, if you do requrie the newly installed golang version as your default, then you can set this environment variables in your shell config. Finally, we can now set the binary as something different because we do not want it to clash with the default golang version. ### Run the binary The binary can be stored in the `/usr/local/bin/` to make any binary available in the system from anywhere. This is not necessary but handy if you are going to use it commonly but don’t need it as the default golang version. ```bash # The binary is stored in the # path_to_cloned_repo/bin # Whatever you want to name the binary cp bin/go /usr/local/bin/go-dev OR cp bin/go /usr/local/bin/go1.24 ``` Once this is done. we can check the installed golang version ```bash go1.24 version ``` With this, you can use it as go1.24 or go-dev as the binary name. So, that is how we install and build from source any golang version above 1.23. ## Conclusion For context, I wanted to check out the latest changes in 1.24, so I cloned the repo and after some trial and error of some commands to build the golang version, I was able to do it correctly. So just decided to share it here, hope you found it helpful. Thank you for reading. Happy Coding :)

Adding SSH Keys for Multiple Accounts in Git

Fri, 22 Mar 2024 00:00:00 UTC

Let's say you have multiple github accounts. One for your personal projects, one for your company that you work at, and one other remote repository account (let's say gitlab). You are juggling with multiple accounts, you should not waste much time and pick a SSH from those remote repository and pull it in your local machine, that makes the process just smooth and saves a ton of time. ### Create a SSH Key To create a SSH key, in linux you can use `ssh-keygen` command. ```bash ssh-keygen -t ed25519 -C "alice@example.com" ``` The above command will prompt you for two things 1. The location where you want to store the key 2. The passphrase for accessing the key ### Add SSH Key to Github Locate to the `ssh` folder and copy the generated `.pub` file to your `github` account. For example, if you have created the key at `~/.ssh/your_name` then copy the contents of the file `~/.ssh/your_name.pub` to your clipbaord. Navigate to your `github` account and in the settings, `SSH and GPG keys` tab, click on `Add SSH key` and copy the contents of your clipboard to the `Key` field. ### Configuring the SSH keys for multiple accounts ```config Host your_company HostName github.com User git IdentityFile ~/.ssh/your_company Host your_name HostName github.com User git IdentityFile ~/.ssh/your_name Host some_name HostName gitlab.com User git IdentityFile ~/.ssh/some_name ``` You can change the `Host` config tag values in the `~/.ssh/conFig` The next time you clone/create a repository on those remote git providers, you need to specify the ssh key for that account. For example, if you have a repository `github.com/StartUp_company/some_wired_project` then you can specify the remote as `git@your_company.com:StartUp_company/some_wired_project`. Here, the `git@your_company` is the `Host` value tag from the `~/.ssh/config`. If that repository is from your `your_company` organisation/user scope, you need to add the `git@your_company` tag, if that's your project, simply add `git@your_name` before the repository url i.e. `your_name/repo_name` which would set the origin as `git@your_name:your_name/repo_name`, here the 1st `your_name` is the tag from the `Host` config and the 2nd `your_name` is the github username. So, in summary if you wanted to use multiple accounts in the same machine, you can understand in the following example: ```bash ssh -T git@your_name git clone https://github.com/your_name/repo_name ``` However, you will need to authenticate with the ssh keys in this way everytime you push/pull a repository. So for that, you can set the origin with the `git@your_name` tag as the host for automatically authenticating the ssh keys on every push/pull or other activities. Thanks for reading, Happy Coding :)

Turn Python dictionary into a neat CSV table

Wed, 20 Mar 2024 00:00:00 UTC

## Populating a Python dict having a table-like structure to a CSV Today, I want to share with you a neat trick I recently discovered for populating a CSV file with data in a table-like structure using Python. ### Writing Key-Value Pairs to a CSV Row Firstly, let's discuss the `write_key_value` function. This function allows us to write key-value pairs to a CSV row. It's particularly useful when dealing with metrics or data that can be represented as simple pairs. ```python # Function to populate a CSV row with key-value pairs def write_key_value(writer, dictionary): for key, value in dictionary.items(): writer.writerow([key, value]) ``` ### Writing a Table-Like Structure to a CSV File Now, let's dive into the `write_table` function, which handles more complex scenarios where the data follows a table-like structure. This function takes into account different types of metrics and adjusts the CSV table structure accordingly. Assuming you have a structure of the dictionary like: ```python data = { "Students": { "John Doe": { "course": "Mathematics", "grade": "A", "attendance": 95, "assignments_completed": 15, "student_id": "JD001" }, "Alice Smith": { "course": "Physics", "grade": "B+", "attendance": 85, "assignments_completed": 12, "student_id": "AS002" }, "Bob Johnson": { "course": "Computer Science", "grade": "A-", "attendance": 90, "assignments_completed": 14, "student_id": "BJ003" } } } ``` And you want to write it to a CSV file, like this: ```csv student, course, grade, attendance, assignments_completed, student_id John Doe, Mathematics, A, 95, 15, JD001 Alice Smith, Physics, B+, 85, 12, AS002 Bob Johnson, Computer Science, A-, 90, 14, BJ003 ``` We can create a function `write_table` that will take in the `dictionary` as the actual data. We want to store the keys of the inner dictionary to be the header/columns of the csv file. As we can see the keys of the inner dict i.e. the value for the key `John Doe` is a dict with the keys `course`, `grade`, `attendance`, etc. which remain the same for the all the keys in the dictionary. So, we can first create a `row_keys` variable to store the keys of the actual dictionary this will be the first column rows in the csv. Further we check if the `row_keys` is a dict and then we append it with the `index_key` which will be the first column in the csv. Since all the keys remain the same for the inner-dict, we can pick the first dict and create the `header` with the inner-dict keys. So, we can write the list `header` to the csv file. Then for each key in the `row_keys` we can create a list `row` with the key and the values of the inner-dict. ```python # Function to populate a CSV with a table-like structure def write_table(writer, dictionary, index_key): row_keys = list(dictionary.keys()) if row_keys and data[row_keys[0]] is not None: headers = [index_key] + list( dictionary[row_keys[0]].keys() ) else: return writer.writerow(headers) for key in row_keys: row = [key] + list(dictionary[key].values()) writer.writerow(row) with open('data.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile) for key in data: write_table(writer, data[key], key) ``` ### Example Usage To illustrate how these functions can be used, let's consider a scenario where we have various types of metrics to populate into a CSV file. We handle key-value paired metrics separately and then populate the CSV with table-like metrics. ```python import csv data = { "Students": { "John Doe": { "course": "Mathematics", "grade": "A", "attendance": 95, "assignments_completed": 15, "student_id": "JD001" }, "Alice Smith": { "course": "Physics", "grade": "B+", "attendance": 85, "assignments_completed": 12, "student_id": "AS002" }, "Bob Johnson": { "course": "Computer Science", "grade": "A-", "attendance": 90, "assignments_completed": 14, "student_id": "BJ003" } }, "Countries": { "USA": { "capital": "Washington, D.C.", "population": 331000000, "area_sq_km": 9833517, "official_languages": ["English", "Spanish"], "currency": "United States Dollar (USD)" }, "India": { "capital": "New Delhi", "population": 1380004385, "area_sq_km": 3287263, "official_languages": ["Hindi", "English"], "currency": "Indian Rupee (INR)" }, "Brazil": { "capital": "Brasília", "population": 212559417, "area_sq_km": 8515770, "official_languages": ["Portuguese"], "currency": "Brazilian Real (BRL)" } } } def populate_table(writer, data, index_key): row_keys = list(data.keys()) if row_keys and data[row_keys[0]] is not None: headers = [index_key] + list(data[row_keys[0]].keys()) else: return writer.writerow(headers) for key in row_keys: row = [key] + list(data[key].values()) writer.writerow(row) with open('data.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile) for key in data: populate_table(writer, data[key], key) writer.writerow([]) ``` ```csv Students,course,grade,attendance,assignments_completed,student_id John Doe,Mathematics,A,95,15,JD001 Alice Smith,Physics,B+,85,12,AS002 Bob Johnson,Computer Science,A-,90,14,BJ003 Countries,capital,population,area_sq_km,official_languages,currency USA,"Washington, D.C.",331000000,9833517,"['English', 'Spanish']",United States Dollar (USD) India,New Delhi,1380004385,3287263,"['Hindi', 'English']",Indian Rupee (INR) Brazil,Brasília,212559417,8515770,['Portuguese'],Brazilian Real (BRL) ``` Thank you, Happy Coding :)

SQLite importing CSV with custom separator

Fri, 15 Mar 2024 00:00:00 UTC

explaination about how i learned how to write a command for importing a csv into a sqlite3 db with cusotm Separator with a single command inline ## Introduction I was exploring some Issues that I can solve given the context of the problem and the intution for the solution. I have some github repositories that I always look for to learn more. Some of them are: - [Steampipe](https://github.com/turbot/steampipe) - [MindsDB](https://github.com/mindsdb/mindsdb) - [Turso](https://github.com/tursodatabase) So, navigating around for a few minutes, I landed on this [issue](https://github.com/tursodatabase/turso-cli/issues/811). The issue is really well explained in terms of what the feature is to be added, how the usage should be, which for a contributor is half the job done. ## What? The [turso-cli](https://github.com/tursodatabase/turso-cli) is a CLI for [Turso](https://github.com/tursodatabase/turso) that is used to manage libsql databases with the [turso](https://turso.tech) platform. This issue is about adding a flag to the `db create` command to allow the user to pass in a custom separator while imporing a csv file into a sqlite3 database. The only puzzle piece left is to answer the question `how`? ## How? So, I went in to the Codebase and found where the `db create` command has been handled and landed on this [file](https://sourcegraph.com/github.com/tursodatabase/turso-cli/-/blob/internal/cmd/db_create.go) While controbuting to open source, I try to do the small steps and solve try to maintain the excitment with that progress. Because if you cannot find the solution, you try to procastinate and in the end nothing gets accomplished. So, breaking down the problem into small chunks is much helpful than solving the entire problem. In this case, I tried to add the flag in the cli which is pretty straight forward. We just add one more entry in the list of flags in the `db create` command. This step is just to add a functional CLI for taking the input of csv-separator string, however, this won't do anything for the functionality part of things, just allow the user to specify the separator/delimitor to use for parsing the CSV file. Currently, the command that `turso db create` uses under the hood for creating a db from a csv file is: ```bash sqlite3 "-csv" "dbName" ".import " ``` The above command is found in the [group flag](https://sourcegraph.com/github.com/tursodatabase/turso-cli/-/blob/internal/cmd/group_flag.go) file. To parse in the separator, we can use another string as `.separator ";"`, here the `;` is the character that should be used as the separator for the csv file into the db. ```bash sqlite3 "-csv" "dbname" ".separator" ";" ".import " ``` Thank you, Happy Coding :)

Golang: Sort Package Introduction

Mon, 15 Jan 2024 00:00:00 UTC

I have been learning Golang for around 2 years now, and I have never paid attention to the sort package, can you believe that! Where was this package hiding? The `sort` package provides convenient methods for sorting slices and user-defined collections in Go. Here's a quick overview of what it can do: ## Sorting Slices To sort a slice of builtin types like ints, strings, etc, you can simply call `sort.Slice()`: ```go nums := []int{5, 2, 6, 3, 1} sort.Slice(nums, func(i, j int) bool { return nums[i] < nums[j] }) // or // sorts.Ints(nums) fmt.Println(nums) ``` ```bash $ go run main.go [1, 2, 3, 5, 6] ``` The sort is in-place, mutating the original slice. You can customize the sort order by providing a less(i, j int) bool function. ## Sorting Struct Slices To sort a slice of custom structs, add a Len(), Less(i, j int) bool, and Swap(i, j int) method: ```go type Person struct { Name string Age int } people := []Person{ {"Bob", 30}, {"John", 20}, {"Alice", 25}, } sort.Slice(people, func(i, j int) bool { return people[i].Age < people[j].Age }) fmt.Println(person) ``` ```bash $ go run main.go [{"Name":"Alice","Age":25},{"Name":"Bob","Age":30},{"Name":"John","Age":20}] ``` This will sort people by age. ## Sorting Maps Maps are inherently unordered in Go. We can't sort them, but we can iterate them in a sorted way. We need a separate slice of keys or values(whicher required), we will sort those keys/values and iterate over them in that order.: ### Sort by Keys To sort a map by keys, we can use the `sort.Strings()` function or any other sort function as per the data structure, there are functions like [sort.Ints](https://pkg.go.dev/sort#Ints), [sort.Float64](https://pkg.go.dev/sort#Float64s), or [sort.Slice](https://pkg.go.dev/sort#Slice). We can create a new slice of keys from the map and then apply the sort on that newly created slice. After the slice of keys is created, we can iterate over it and access the map values in a order of sorted keys. ```go counts := map[string]int{ "hello": 5, "world": 2, "foo": 3, } keys := make([]string, 0, len(counts)) // extract keys for k := range counts { keys = append(keys, k) } // sort keys sort.Strings(keys) // iterate with the sorted keys slice for _, k := range keys { fmt.Println(k, counts[k]) } ``` ```bash $ go run main.go foo 3 hello 5 world 2 ``` This prints the map ordered by key. You can sort by value too. ### Sort by Value To iterate the map with sorting order of values, we can approach it in a similar way. We create a slice of keys. This time, we don't sort the keys, instead, we change the position of the slice of keys depending on the values. This changes the key order based on the sorted values in the map. By similarly iterating over the key slice, we iterate over the map in a sorted order of the values. ```go counts := map[string]int{ "hello": 5, "world": 2, "foo": 3, } keys := make([]string, 0, len(counts)) // extract keys for k := range counts { keys = append(keys, k) } // sort by value // i.e. change the order of key based on values sort order sort.SliceStable(keys, func(i, j int) bool { return counts[keys[i]] < counts[keys[j]] }) // iterate sorted for _, k := range keys { fmt.Println(k, counts[k]) } ``` ```bash $ go run main.go world 2 foo 3 hello 5 ``` The `sort.SliceStable` function is used to sort the slice in place. It takes in a slice and the less function which is the actual logic for comparison in the values. This function sorts the key elements based on the values in the map, and thereby the key slice is shuffled by the sorted order of the values in the map. The sort package is super useful for quickly organizing Go data structures. There are [Find](https://pkg.go.dev/sort#Find) which tries to return a index of the first element that satisfies a comparison condition with a flag or found or not with a boolean. There is also a [Search](https://pkg.go.dev/sort#Search) which is used specifically for searching elements in a sorted array, it also gives a first index of the occuring element. But that is a topic for another day! Thank you, Happy Coding :)

Golang: Test Output JSON

Mon, 01 Jan 2024 00:00:00 UTC

I just discovered that we can generate a JSON output of test results in Golang. I found this [here](https://youtu.be/cf72gMBrsI0?t=80). Let's take a fresh simple example. ```go package jsontest func hello() string { return "Hello, World!" } ``` ```go package jsontest import ( "testing" ) func TestHello(t *testing.T) { want := "Hello, World!" got := hello() t.Logf("got: %q", got) if got != want { t.Errorf("got: %q, want: %q", got, want) } } ``` Here, we have a function `hello` that simply returns a string, the `TestHello` function in the `jsontest` package will check if the returned string is correctly returned or not. So, we can test this with `go test ./...` command, this will give out the output in a standard output/error in text format. However, if we add the `-json` flag, we can get the output in JSON format. ```bash go test ./... -json ``` ```json {"Time":"2024-01-01T20:52:45.861974085+05:30","Action":"start","Package":"json-test"} {"Time":"2024-01-01T20:52:45.863133332+05:30","Action":"run","Package":"json-test","Test":"TestHello"} {"Time":"2024-01-01T20:52:45.863142397+05:30","Action":"output","Package":"json-test","Test":"TestHello","Output":"=== RUN TestHello\n"} {"Time":"2024-01-01T20:52:45.863148346+05:30","Action":"output","Package":"json-test","Test":"TestHello","Output":" main_test.go:10: got: \"Hello, World\"\n"} {"Time":"2024-01-01T20:52:45.863151351+05:30","Action":"output","Package":"json-test","Test":"TestHello","Output":" main_test.go:12: got: \"Hello, World\", want: \"Hello, World!\"\n"} {"Time":"2024-01-01T20:52:45.863157014+05:30","Action":"output","Package":"json-test","Test":"TestHello","Output":"--- FAIL: TestHello (0.00s)\n"} {"Time":"2024-01-01T20:52:45.863160418+05:30","Action":"fail","Package":"json-test","Test":"TestHello","Elapsed":0} {"Time":"2024-01-01T20:52:45.863411555+05:30","Action":"output","Package":"json-test","Output":"FAIL\n"} {"Time":"2024-01-01T20:52:45.863438344+05:30","Action":"output","Package":"json-test","Output":"FAIL\tjson-test\t0.001s\n"} {"Time":"2024-01-01T20:52:45.863443461+05:30","Action":"fail","Package":"json-test","Elapsed":0.001} ``` Pretty cool right? This is really useful for programmatically taking the output and parsing it to get the metrics. We can even combine with the coverage flag to get the coverage metrics as well. ```bash go test ./... -json -cover ``` ```json {"Time":"2024-01-01T21:13:30.771976961+05:30","Action":"start","Package":"jsontest"} {"Time":"2024-01-01T21:13:30.775118482+05:30","Action":"run","Package":"jsontest","Test":"TestHello"} {"Time":"2024-01-01T21:13:30.775172535+05:30","Action":"output","Package":"jsontest","Test":"TestHello","Output":"=== RUN TestHello\n"} {"Time":"2024-01-01T21:13:30.775201647+05:30","Action":"output","Package":"jsontest","Test":"TestHello","Output":" main_test.go:10: got: \"Hello, World!\"\n"} {"Time":"2024-01-01T21:13:30.775231759+05:30","Action":"output","Package":"jsontest","Test":"TestHello","Output":"--- PASS: TestHello (0.00s)\n"} {"Time":"2024-01-01T21:13:30.775253928+05:30","Action":"pass","Package":"jsontest","Test":"TestHello","Elapsed":0} {"Time":"2024-01-01T21:13:30.775269402+05:30","Action":"output","Package":"jsontest","Output":"PASS\n"} {"Time":"2024-01-01T21:13:30.776153185+05:30","Action":"output","Package":"jsontest","Output":"coverage: 100.0% of statements\n"} {"Time":"2024-01-01T21:13:30.776808599+05:30","Action":"output","Package":"jsontest","Output":"ok \tjsontest\t0.004s\tcoverage: 100.0% of statements\n" {"Time":"2024-01-01T21:13:30.777814589+05:30","Action":"pass","Package":"jsontest","Elapsed":0.006} ``` I am planning to use this in my workflow for integrating the output of the tests suite with specific tests. For running the specific tests, you can use `go test -run TestName` command, this will only run the provided test function. ```go // main.go package jsontest func hello() string { return "Hello, World!" } func add(x, y int) int { return x + y } ``` ```go // main_test.go package jsontest import ( "testing" ) func TestHello(t *testing.T) { want := "Hello, World!" got := hello() t.Logf("got: %q", got) if got != want { t.Errorf("got: %q, want: %q", got, want) } } func TestAdd(t *testing.T) { want := 2 got := add(1, 1) t.Logf("got: %d", got) if got != want { t.Errorf("got: %d, want: %d", got, want) } } ``` So, we have two test in this go module, we can run a specific test using `go test -run TestName` command as so: ```bash go test -run TestAdd -json ``` ```json {"Time":"2024-01-01T21:33:44.19397581+05:30","Action":"start","Package":"jsontest"} {"Time":"2024-01-01T21:33:44.198067398+05:30","Action":"run","Package":"jsontest","Test":"TestAdd"} {"Time":"2024-01-01T21:33:44.198150156+05:30","Action":"output","Package":"jsontest","Test":"TestAdd","Output":"=== RUN TestAdd\n"} {"Time":"2024-01-01T21:33:44.198197444+05:30","Action":"output","Package":"jsontest","Test":"TestAdd","Output":" main_test.go:19: got: 2\n"} {"Time":"2024-01-01T21:33:44.198217057+05:30","Action":"output","Package":"jsontest","Test":"TestAdd","Output":"--- PASS: TestAdd (0.00s)\n"} {"Time":"2024-01-01T21:33:44.198230965+05:30","Action":"pass","Package":"jsontest","Test":"TestAdd","Elapsed":0} {"Time":"2024-01-01T21:33:44.198241628+05:30","Action":"output","Package":"jsontest","Output":"PASS\n"} {"Time":"2024-01-01T21:33:44.19869148+05:30","Action":"output","Package":"jsontest","Output":"ok \tjsontest\t0.004s\n"} {"Time":"2024-01-01T21:33:44.198822637+05:30","Action":"pass","Package":"jsontest","Elapsed":0.005} ``` As we can see, there is only one test being executed and the output of the test is in JSON format. These are really good flags and options to have as they make the output more portable. I will be planning to use this to improve my workflow in testing and developing open source projects and personal projects as well. I am really inspired by the Teej's video of executing anything in NeoVim.

LibSQL: Query a remote Turso database with cURL

Wed, 15 Nov 2023 00:00:00 UTC

If you are using a local [libsql](https://turso.tech/libsql) database, it is quite easy to query the database, but for a remote or a database on a hosted cloud platform like [turso](https://turso.tech/), we can use other clients or the api itself to query the data. We can use the turso cli to get the authentication token for the database and then query the database. ## Turso CLI Using the [turso-cli](https://docs.turso.tech/reference/turso-cli) to access the turso platform. We will use turso cli to create a libsql database, create authentication tokens, and query the db. ### Create a database (it's optional, you might already have a database) ```bash turso db create ``` You will get a database on the turso cloud platform with some random interesting name like a passphrase. Use the command `turso db list` and copy the URL ```graphql DB_URL=dbname-orgname.turso.io ``` ### Create an authentication token for a particular database ```bash turso db tokens create db_name ``` Copy the JWT token and this will be used as a authentication token when accessing the remote database in the turso cloud. ```bash TOKEN=abcdef.12345.wxyz DB_URL=dbname-orgname.turso.io ``` * Querying the database using curl or other [api clients](https://docs.turso.tech/libsql/client-access) ```bash curl -s -H "Authorization: bearer $TOKEN" \ -d '{"statements": ["SELECT name FROM sqlite_master WHERE type=\"table\";"]}' \ $DB_URL ``` We can use `curl` or any api client tools to send queries to the database hosted on the turso platform. We need to provide the JWT token in the `Authorization` header to connect to that particular database. The request's body is a JSON string with a list of statements to query the database. ```graphql [ { "results": { "columns": ["name"], "rows":[ ["libsql_wasm_func_table"], ["_litestream_seq"], ["_litestream_lock"], ["sqlite_sequence"], ["user"] ] } } ] ``` The result is a list of key-value pairs as `columns` and `rows` for each of the statements in the body. The columns are a list of column names requested in the query, and the rows are a list of rows where each row is a list of field values from the query.

Read a Rss Feed with a URL in Golang

Fri, 11 Nov 2022 00:00:00 UTC

## Reding Rss Feed We can use golang's [encoding/xml](https://pkg.go.dev/encoding/xml) package to read a Rss feed. Though we have to be speicific of what type of structure the Rss feed has, so it is not dynamic but it works really well with structs. I have covered a few nuances of reading XML file in the [config file reading](https://www.meetgor.com/golang-config-file-read/#reading-xml-file) post of the 100 days of golang series. ### Get request to Rss feed We first need to send a `GET` request to the Rss feed, we can use the [http](https://pkg.go.dev/net/http) package to grab the response. ```go package main import ( "log" "net/http" ) func main() { url := "https://meetgor.com/rss.xml" response, err := http.Get(url) if err != nil { log.Fatal(err) } log.Println(response.Body) defer response.Body.Close() } ``` So, in the above example, we have used the `net/http` package to send a `GET` request with the [Get](https://pkg.go.dev/net/http#Get) funciton. The function takes in a string as a `URL` and returns either the object as response or an error. If there arose any error, we simply exit out of the program and log the error. If the error is `nil`, we return the response in the `response` variable. This builds up a good foundation for the next step to read the response body and fetching the actual bytes from the response. ### Fetch the content from the Link Since we have the `response` object, we can use the [io.ReadAll](https://pkg.go.dev/io#ReadAll) function to read the bytes in the response body. The function takes in the [Reader](https://pkg.go.dev/io#Reader) object in this case it is [ReadCloser](https://pkg.go.dev/io#ReadCloser) object as a http object. The function then returns the slice of bytes/int8. The slice then can be interpreted as string or other form data that can be used for parsing the xml from the response. ```go package main import ( "log" "net/http" "io" ) func main() { url := "https://meetgor.com/rss.xml" response, err := http.Get(url) if err != nil { log.Fatal(err) } data, err := io.ReadAll(response.Body) if err != nil { log.Fatal(err) } log.Println(string(data)) log.Printf("Type -> %T", data) } ``` ``` ... ... ... Type -> []uint8 ``` So, we can see that the parsed content is indeed xml, it is type casted to string from the slice of bytes. This can be further used for the parsing the text as Rss structure and fetch the required details. ## Parsing Rss with a struct We can now move into creating a struct for individual tags required in the parsing. ```go package main import ( "encoding/xml" "io" "log" "net/http" ) type Rss struct { XMLName xml.Name `xml:"rss"` Channel Channel `xml:"channel"` } type Channel struct { XMLName xml.Name `xml:"channel"` Title string `xml:"title"` Description string `xml:"description"` Item []Item `xml:"item"` } type Item struct { XMLName xml.Name `xml:"item"` Title string `xml:"title"` Link string `xml:"link"` } func main() { url := "https://meetgor.com/rss.xml" response, err := http.Get(url) if err != nil { log.Fatal(err) } data, err := io.ReadAll(response.Body) if err != nil { log.Fatal(err) } log.Println(string(data)) } ``` If you would look at the [rss feed](https://meetgor.com/rss.xml), you can see it has a structure of tags and elements. The `rss` tag is the root tag, followed by `channel` and other types of nested tags speicific for the type of information to be stored like `title` for the title in the feed, `link` for the link to the feed, etc. So, we create those as structure, the root structure is the `Rss` which we will create with a few attributes like `Channel` and the name of the current tag. In the `Rss` case the name of the tag/element is `rss`, so it is given the `xml.Name` as `xml:'rss'` in backticks indicating the type hint for the field. The next field is the `Channel` which is another type(custom type struct). We have defined `Channel` as a struct just after it that will hold information like the `title`, `description` of the website. We also have the `xml.Name` as `xml:"channel"` which indicates the current struct is representation of `channel` tag in the rss feed. Finally, we also have a custom type struct as `Item`. The `Item` struct has a few attributes like `Title`, `Link` and you can now start to see the pattern, you can customize it as per your requirements and speicifications. ```go package main import ( "encoding/xml" "io" "log" "net/http" ) type Rss struct { XMLName xml.Name `xml:"rss"` Channel Channel `xml:"channel"` } type Channel struct { XMLName xml.Name `xml:"channel"` Title string `xml:"title"` Description string `xml:"description"` Item []Item `xml:"item"` } type Item struct { XMLName xml.Name `xml:"item"` Title string `xml:"title"` Link string `xml:"link"` } func main() { url := "https://meetgor.com/rss.xml" response, err := http.Get(url) if err != nil { log.Fatal(err) } data, err := io.ReadAll(response.Body) if err != nil { log.Fatal(err) } // New Code d := Rss{} err = xml.Unmarshal(data, &d) if err != nil { log.Fatal(err) } for _, item := range d.Channel.Item { log.Println(item.Title) } } ``` ``` $ go run main.go Why and How to make and use Vim as a text editor and customizable IDE Setting up Vim for Python Setting up Vim for BASH Scripting Vim: Keymapping Guide ... ... ... Django + HTMX CRUD application PGCLI: Postgres from the terminal Golang: Closures Golang: Interfaces Golang: Error Handling Golang: Paths Golang: File Reading Golang: JSON YAML TOML (config) File Reading. ``` So, here we have initialized the `Rss` struct as empty and then used the [Unmarshal](https://pkg.go.dev/encoding/xml#Unmarshal) method in the `xml` package. The Unmarshal method will parse the data as per the type of either int, float, bool or string, any other type of data will be discarded as interface or struct. We can usually parse any valid type of data into `Unmarshal` method and it generally gives a proper expected outcome. The Unmarshal method takes in the slice of byte and the second paramter as pointer to a struct or any variable that will store the parsed xml content from the slice of byte. The function just returns the error type, either `nil` in case of no errors, and returns the actual error obejct if there arise any type of error. So we parse the `data` which is a slice of byte to the funciton and the reference to the `d` object which is a empty `Rss` object. This will get us the data in the `d` object. We can then iterate over the object as per the struct and use the perform operations like type casting or converting types, etc to get your required data back. In the above example, we simply iterate over the `d.Channel.Item` which is a list of elements of tag `item` in the rss feed. Inside the for loop, we can access the object and simply print or perform any sort of operations. I have simply printed the list of articles with titles. Links for the code available on the [100 days of golang](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/scripts/files/read/config_files/xml/rss.go) GitHub repository. So, that's how we parse an XML feed in golang. Just plug and play if you have a similar type of structure of the Rss XML feed. Happy Coding :)

Create a Non-Clustered Index in Django with Postgres as DB

Thu, 10 Nov 2022 00:00:00 UTC

## What is a non-clustered index? A non-clustered index is a seperate structure than an actual table in the database, it stores the non-clustered index key(the column which we want to sort in the table), and a pointer to the actual values based on the index key. So, non-clustered indexes do not change the physical order of the table records, instead it holds a structure that can provide a easier and distinct way to fetch objects based on a particular column as the primary key in the structure. ## How to create a non-clustered index in django In django, we can use the [db_index](https://docs.djangoproject.com/en/4.1/ref/models/indexes/) property on a field(s) to create a index on the table/model. ### Add the property to the field in the model Chose a field in which, you want to add a index. It can be a foreign key or any other normal field defined in your model. We have used the typical blog model, so used in the some of my [TILS](https://www.meetgor.com/tils/) in django, it is just convenient to explain and understand as well. We have a django project named `core` and it has a app `blog` with a model defined below. The model `Article` has a few attributes like `title`, `description`, `content` and `status`. ```python from django.db import models ARTICLE_STATUS = [ ("PUBLISHED", "Published"), ("DRAFT", "Draft"), ] class Article(models.Model): title = models.CharField(max_length=128, db_index=True) description = models.CharField(max_length=512) content = models.TextField() status = models.CharField(max_length=16, choices=ARTICLE_STATUS, default="DRAFT") def __str__(self): return self.title ``` So, we have added a `db_index` to the title column in the model as a property. This will be equivalent to creating a index in `SQL` as follows: ``` $ python manage.py makemigrations Migrations for 'blog': blog/migrations/0002_alter_article_title.py - Alter field title on article ``` ``` $ python manage.py migrate Operations to perform: Apply all migrations: admin, auth, blog, contenttypes, sessions Running migrations: Applying blog.0002_alter_article_title... OK ``` Indexes are not standard as in SQL, but each vendor(sqlite, postgres, mysql) have their own flavour of syntax and naunces. ```sql CREATE INDEX "blog_article_title_3c514952" ON "blog_article" ("title"); CREATE INDEX "blog_article_title_3c514952_like" ON "blog_article" ("title" varchar_pattern_ops); ``` The above index commands are specific to the field, as the title field is a varchar, it has two types of index, it can generate one with simple match and other for `LIKE` comparisons because of string comparison behaviour. So, we just created a simple index and now if we query the db for a particular `title` which now has its own index for the table `blog_article`. This means, we will be able to fetch queries quickly if we are specifically filtering for `title`. ### Adding some data records We can add a few data records to test the query from the databse, you can ignore this part as it would be just setting up a django project and adding a few records to the databse. This part won't make sense for people reading to get the actual stuff done, move to the next part please. ``` python manage.py createsuperuser # Create a super user and run the server python manage.py runserver # Locate to http://127.0.0.1:8000/admin # Create some records in the artilce model ``` So, after creating some records, you should have a simple database and a working django application. ```sql SELECT * FROM blog_article; ``` ``` blog_test=# SELECT * FROM blog_article; id | title | description | content | status ----+----------+-------------+---------------------------+----------- 1 | test | test 1 | test content | DRAFT 2 | testpost | test 2 | test content more content | DRAFT 3 | newpost | test 3 | test nothing | PUBLISHED (3 rows) ``` ## Testing Queries We can now use SQL queries or django filters to check if we get results by a sequential or an index scan. If we have a filter of `title` we will get the results after performing an `Index Scan` which means, it will look up in the index columns rather than scanning the entire table of records. This is a way **we can test the indexes are working, efficiency is a differnet topic.** We can't get a idea of performance with this little data and just one connection. A real time database and having multiple conncurrent requests and connections is a good environment to test(don't do it in a production db :) ```sql EXPLAIN SELECT * FROM blog_article WHERE description LIKE 'test 2'; ``` ``` blog_test=# EXPLAIN ANALYSE SELECT * FROM blog_article WHERE description LIKE 'test'; --------------------------------------------------------------------------------------------------------- Seq Scan on blog_article (cost=0.00..11.00 rows=1 width=880) (actual time=0.180..0.181 rows=0 loops=1) Filter: ((description)::text ~~ 'test'::text) Rows Removed by Filter: 3 Planning Time: 0.189 ms Execution Time: 0.217 ms (5 rows) ``` The above query selects the records whose `description` is like `test 2`, this performs a `Sequenitial Scan` in the database i.e. iterating over the records one by one of the order of the primary key / id of the records in the table. ```sql EXPLAIN SELECT * FROM blog_article WHERE title LIKE 'test 2'; ``` ``` blog_test=# EXPLAIN ANALYSE SELECT * FROM blog_article WHERE title LIKE 'test'; --------------------------------------------------------------------------------------------------------- Index Scan using blog_article_title_3c514952_like on blog_article (cost=0.14..8.16 rows=1 width=880) (actual time=0.043..0.048 rows=1 loops=1) Index Cond: ((title)::text = 'test'::text) Filter: ((title)::text ~~ 'test'::text) Planning Time: 0.208 ms Execution Time: 0.093 ms (5 rows) ``` In the above query, the select statement has a filter with the title being like `test 2`, and since we have a index for looking for like of title column, the database performs a index scan on that table and fetches the result. Here are some tradeoffs, the planning is more and the execution time is less, this is quite logical as it would take time to make decision because the database has more options than before creating indexes. In the query where we filtered the description, the planning time was less as it makes sense there was just one option to go for sequential scan, but it took time to perform the operation as it would scan the entire table one by one. ## Using Django to test queries We can even use django to filter out the objects in the table. We simply use the `filter` method to check with a particular value. We can use the shell, to perform some queries. You can use this in your views or viewsets as per your requirements and constraints. We can even use `explain` to see what the underlying `sql` got executed out from the ORM. The [explain](https://docs.djangoproject.com/en/3.1/ref/models/querysets/#explain) function is similar to the `EXPLAIN ANALYSE` command in the `sql` queries. It gives a bit of context on how the query was executed. ``` $ python manage.py shell ``` ```python >>> from blog.models import Article >>> Article.objects.filter(description='test 1') ]> >>> Article.objects.filter(description='test 1').explain() "Seq Scan on blog_article (cost=0.00..11.00 rows=1 width=880)\n Filter: ((description)::text = 't est 1'::text)" >>> Article.objects.filter(title='test') ]> >>> Article.objects.filter(title='test').explain() "Index Scan using blog_article_title_3c514952_like on blog_article (cost=0.14..8.16 rows=1 width=8 80)\n Index Cond: ((title)::text = 'test'::text)" ``` We can use `__contains` for replicating the behaviour of `LIKE` in python/django from SQL. The below example will check if the title has a word `test` in any records of the database. ``` >>> Article.objects.filter(title__contains='test') , ]> ``` BONUS: We can even get the underlying SQL with the `.query.__str__()` method. ``` articles = Article.objects.filter(title__contains='test') articles.query.__str__() ``` ``` 'SELECT "blog_article"."id", "blog_article"."title", "blog_article"."description", "blog_article"." content", "blog_article"."status" FROM "blog_article" WHERE "blog_article"."title"::text LIKE %test %' ``` Here, we are able to see that clearly, that the django orm used the `LIKE` clause for comparing the title. Further readings and references: - [Indexing in Postgres](https://medium.com/geekculture/indexing-in-postgres-db-4cf502ce1b4e) - [Indexing refernece Django docs](https://docs.djangoproject.com/en/4.1/ref/models/indexes/) - [Non-Clustered indexing](https://gudevsoc.com/what-is-non-clustered-index-in-sql-with-example/)

Django Bulk Update QuerySet objects

Mon, 31 Oct 2022 00:00:00 UTC

Let's say, I have a lots of objects which I want to update with a particular field or fields. We can use the [bulk_update](https://docs.djangoproject.com/en/4.1/ref/models/querysets/#bulk-update) method with the model name. ```python # blog/models.py from django.db import models ARTICLE_STATUS = [ ("PUBLISHED", "Published"), ("DRAFT", "Draft"), ] class Article(models.Model): title = models.CharField(max_length=128) description = models.CharField(max_length=512) content = models.TextField() status = models.CharField(max_length=16, choices=ARTICLE_STATUS, default="DRAFT") def __str__(self): return self.title ``` Let's say we have a simple model `Article` with a few typical attributes like `title`, `description`, `content`, and `status`. We have the status as a choice field from two options as `Draft` and `Published`. It could be a boolean field, but that looks too gross for a article status. ```python from blog.models import Article articles = Article.objects.filter(status="draft") for i in range(len(articles)): articles[i].status = "published" Article.objects.bulk_update(articles, ["status",]) ``` In the above code, the `Articles` model is filtered by the status of `draft`. We iterate over the `QuerySet` which will contain the objects of the articles, by setting the object's properties to the value we want to set. We are jsut setting the value of the property of the object for each object. This just makes a changes to the `QuerySet`, by using the `bulk_update` method, the two parameters required are the `QuerySet` and the list of `fields` which are to be updated. The function returns the number of objects/records updated. ```python >>> from blog.models import Article >>> articles = Article.objects.filter(status="DRAFT") >>> articles , ]> >>> for i in range(len(articles)): ... articles[i].status = "PUBLISHED" ... >>> articles , ]> >>> >>> Article.objects.bulk_update(articles, ['status',]) 2 >>> Article.obejcts.get(title="test 1").status 'PUBLISHED' >>> Article.objects.filter(status="DRAFT") >>> ``` As, we can see here there were two obejcts `test 1` and `test 2` objects with the status as `Draft`. By iterating over the queryset and assigning the status of the object to published, the query set was changed and modified locally. By using the `bulk_update` method, we parsed the queryset and the list of attributes to be updated into the function. This gives us the number of objects which were updated, in this case `2`. We then look into the article actual record in the database and it has indeed updated to the value we set in this operation.

Python Pipreqs: Generate requirements file from the imported packages

Wed, 14 Sep 2022 00:00:00 UTC

## Introduction [Pipreqs](https://pypi.org/project/pipreqs/) is a python package that allows us to list all the pacakges which are imported in a python project. This is a great package for reducing the amount of redundant packages for a project. ## Install pipreqs You can install pipreqs with one of the many ways with pip, pipx, or any other pacakge management tool. I personally use pipreqs with `pipx` as it remains isolated from the rest of my project dependencies. ### Using simple pip install We can install with pip by creating a virtual environment or in a existing virtual environment. ``` pip install virtualenv venv source venv/bin/activate pip install pipreqs ``` ### Using pipx We can install pipreqs with pipx. [Pipx](https://pypi.org/project/pipx/) is also a python package but used as a tool to install any cli specific tool with the isolated environment. ``` pipx install pipreqs pipx run pipreqs ``` ## Using pipreqs We need to specify the encoding, which is used for reading the files while capturing the imports from the project. ``` pipx run pipreqs --encoding=utf-8 . ``` Additionaly, we can specify the `path` or filename where it will be used to save the imported packages. The `--savepath` option takes in the path to the file where you want to generate the list of the packages to be installed. ``` pipx run pipreqs --encoding=utf-8 --savepath reqs.txt . ``` Though this doesn't guarentee all the requirements for a file, it is really helpful for explicitly used packages in the python project.

Django: Get list of all models and associated fields in a django project

Tue, 02 Aug 2022 00:00:00 UTC

## Context Let's say we want the list of all the models and associated attributes in all the applications of a django project, we can do that using the [django.apps](https://docs.djangoproject.com/en/4.0/ref/applications/) with apps method. ## Get all the models in a project To fetch all the models, we can use the [get_models](https://docs.djangoproject.com/en/4.0/ref/applications/#django.apps.AppConfig.get_models) methods, it will return a list of model classes in all the entire project(all applications). We can import all the models in the django project with the command: ```python from django.apps import apps models = apps.get_models() ``` ``` [, , , , , , , , , , , , , , , ] ``` We are importing the apps and creating a list of the models in our django project. The Django app command will load all the applications in the project, and the [get_models]() method will fetch the associated models. This has resulted in a list of model class objects, we can iterate over them and fetch the required details, we want. For instance, If I am interested in the name of these models, I can use the `__name__` property to fetch the model's name. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(model.__name__) ``` ``` LogEntry Permission Group ContentType Session EmailAddress EmailConfirmation SocialApp SocialAccount SocialToken TimeStampedModel User Tags Series Article Blog ``` So, from the above example, we can see we have accessed all the model names in our entire django project. ## Access Application name associated with a model For accessing the name of the application from the model class, we can use the `_meta` attribute followed by the `app_label` property to get the `app_name` associated with the model. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(f"{model._meta.app_label} -> {model.__name__}") ``` ``` admin -> LogEntry auth -> Permission auth -> Group contenttypes -> ContentType sessions -> Session account -> EmailAddress account -> EmailConfirmation socialaccount -> SocialApp socialaccount -> SocialAccount socialaccount -> SocialToken user -> TimeStampedModel user -> User articles -> Tags articles -> Series articles -> Article blog -> Blog ``` In the above example, we can see we have printed all the models with their associated application names. ## Accessing all the attributes associated with a model To access all the fields/property/attributes associated with a model, we can again use the `_meta` attribute followed by the `get_fields` method. This method will return a list of field objects. For accessing the name of those attributes/fields, we have to iterate over the list and then further use `name` property. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(model.__name__) field_list = model._meta.get_fields() for field in field_list: print(field.name) ``` ``` LogEntry id action_time user content_type object_id object_repr action_flag change_message Permission group user id ... ... Blog article id name description authors ``` So, that is how we get all the associated field names in the associated models in our django projects. Also, there are a lot of attributes, we can access with the apps property. The `__dict__.keys()` can be used to get the list of all associated properties or other methods in a class instance. ``` >>> m[14]._meta.get_fields()[4].__dict__.keys() dict_keys(['name', 'verbose_name', '_verbose_name', 'primary_key', 'max_length', '_unique', 'blank', 'null', 'remote_field', 'is_relation', 'default', 'editable', 'serialize', 'unique_for_date', 'unique_for_month', 'unique_for_year', 'choices', 'help_text', 'db_index', 'db_column', '_db_tablespace', 'auto_created', 'creation_counter', '_validators', '_error_messages', 'error_messages', 'db_collation', 'validators', 'attname', 'column', 'concrete', 'model']) ``` In the above example, I am using a list of models and getting the list of all the attributes associated with a field of a model. This can be applied and other properties can be accessed. ## Get Models with a specific app Let's say we want all the models associated with a particular application in the project, we can do that by specifying the name of the application. ```python from django.apps import apps app_info = apps.get_app_config('articles') print(app_info.__dict__.keys() print(app_info.verbose_name) print(app_info.models) print(app_info.models['article'].__dict__.keys()) ``` ``` dict_keys(['name', 'module', 'apps', 'label', 'verbose_name', 'path', 'models_module', 'models']) 'Article' {'tags': , 'series': , 'article': } dict_keys(['__module__', 'Article_Status', '__str__', 'get_absolute_url', '__doc__', '_meta', 'DoesNotExist', 'MultipleObjectsReturned', 'title', 'description', 'content', 'status', 'get_status_display', 'blog_id', 'blog', 'author_id', 'author', 'timestampedmodel_ptr_id', 'timestampedmodel_ptr']) ``` So, we can see that we have got the information about the app `articles` in the proejct where we can get the `verbose_name` property to fetch the human-readable format of the article model. Further, we can get all the models associated with the `articles` application. We get back a dict with the model name as the key and the class reference as the value. We have accessed the `article` model in the `articles` application and fetched all the associated properties or methods in the model. For further references, you can visit the [django apps documentation](https://docs.djangoproject.com/en/4.0/ref/applications/) to get more relevant methods and properties.

Django Blog DevLog: Load Frontmatter data into Template/Model Form Fields

Mon, 01 Aug 2022 00:00:00 UTC

## Introduction I have an Article Form where I load my post into it directly, it might have frontmatter. So what I wish to achieve is when I paste in the content, the frontmatter should be picked up and it should render the form fields like `title`, `description`, and then also remove the frontmatter from the content. To do that, we will require a model to work with and a form based on that model. We will exclude a few fields from that model so as to process these attributes on the server side. I am working on my Blog project which is a simple Django application. You can get the source code for the project on the [GitHub repository](https://github.com/mr-destructive/techstructive-blog/). ## Django Project Context The techstructive-blog is a django project, which has a couple of applications currently, not in a good situation. There are apps like `article`, `blog`, and `user`. This project has templates and static folder in the base directory. The project is deployed on [railway](https://www.railway.app) this is an always under development project, you can check out the [techstructive-blog](https://django-blog.up.railway.app). We can get the bits and pieces of the project details required for understanding what I want to do with the following sections. ### Article Model We have an `Article` model with attributes like `title`, `description`, `author` as a Foreign Key to the user model, and a few other attributes which is not related to what we are trying to achieve right now or at least don't require an explanation. We have a model method called `get_absolute_url` for getting a URL in order to redirect the client when the model instance is created or updated from the backend. You can definitely check out the details of these small components or templates in the project repository. ```python # articles/models.py class Article(TimeStampedModel): class Article_Status(models.TextChoices): DRAFT = ( "DRAFT", _("Draft"), ) PUBLISHED = ( "PUBLISHED", _("Published"), ) title = models.CharField(max_length=128) description = models.CharField(max_length=256) content = models.TextField(default="", null=False, blank=False) status = models.CharField( max_length=16, choices=Article_Status.choices, default=Article_Status.DRAFT, ) blog = models.ForeignKey(Blog, on_delete=models.CASCADE, null=True, blank=True) author = models.ForeignKey(User, on_delete=models.CASCADE, related_name="author") def __str__(self): return self.title def get_absolute_url(self): return reverse('articles:article-detail', args=[str(self.id)]) ``` In the below snippet, we have the forms defined in the article application for creating or updating of article instance. We will be using model forms as our form data should contain fields related to a model in this case the `Article` model. So when we inherit the `forms.ModelForm` in our custom `ArticleForm` we simply need to specify the model and it will take in all the attributes of that model by default, but if we specify the `fields` or `exclude` tuples, it will include only or exclude only the provided list of attributes from the model. We have also added the widgets for the form fields which will allow us to customize the way the fields in the template/form will render. We can specify the HTML attributes like `width`, `height`, `style`, etc. ### Article Form ```python # article/forms.py from django import forms from .models import Article class ArticleForm(forms.ModelForm): class Meta: model = Article exclude = ( "created", "updated", "author", ) widgets = { "title": forms.TextInput( attrs={ "class": "form-control", "style": "max-width: 450px; align: center;", "placeholder": "Title", } ), "description": forms.TextInput( attrs={ "class": "form-control", "style": "max-width: 900px; ", "placeholder": "Description", } ), "content": forms.Textarea( attrs={ "class": "form-control post-body", "id": "text-content", "style": "max-width:900px;", "hx-post": "/article/meta/", "placeholder": "Content", } ), "blog": forms.Select( attrs={ "class": "form-control", "placeholder": "Blog Publication", } ), } ``` So, these are my models and form files in the article app. Using htmx and without any javascript I want to update the form so that it picks up the front matter in the content field which is a text area and fills the title, description other attributes automatically for me. This can be done in a lot of ways, but I will be sharing one of the ways that I recently used in my blog project. This process involves creating a class-based view and adding a `POST` method that won't post any data to the backend but will send necessary data to the view. Let's see the process before diving into any of the code: ## Gist of the Process - Attach a `hx-post` attribute to the form field for sending the request to a view - When the request is sent, the data is loaded with `request.POST`, it is cleaned and converted in python-readable format with json. - Once we have the data, we try to use the `frontmatter.loads` function that will load the content and if we have a frontmatter in the text, it will load it as a `frontmatter.POST` object. - We will extract `title`, `description`, and other data fields from the object. - We will initialize a Form instance of Article, with the initial data values as the extracted data from the frontmatter. - Now we have two options: - If the article instance already exists i.e. we are updating the article - Else we are creating a new article Accordingly, we will load the form in the respective templates i.e. `update.html` for updating the existing articles and `article-form.html` for a new article. ## Adding HTMX Magic If you'd have seen we have a `hx-post` attribute in the `article/forms.py` file, the `content` widget has a `hx-post` attribute which sends a post request to the `article/meta/` URL route. This route we will bind to the `ArticleMetaView` which we will define in a few moments. This is usually sent once we change a certain text in the content field, so we can modify it as per your requirement with `hx-trigger` that can enable us to specify the trigger event or the type of trigger we want. Further, you can read from the [htmx docs](https://htmx.org/docs/#trigger-modifiers) about these triggers and other attributes. ```python # article/urls.py from django.urls import path from . import views app_name = "articles" urlpatterns = [ path("", views.ArticleCreateView.as_view(), name="article-create"), path("/", views.ArticleDetailView.as_view(), name="article-detail"), path("delete//", views.ArticleDetailView.as_view(), name="article-delete"), path("edit/", views.ArticleDetailView.as_view(), name="article-update"), # the new view that we will create path("meta/", views.ArticleMetaView.as_view(), name="article-meta"), ] ``` ## Capture Frontmatter Meta-data View Along with the Create, Detail/List, Update, Delete View, I will create a separate class called `ArticleMetaView` that will fetch the form fields and render the templates again but this time it will fill in the frontmatter meta-data in the fields if the content is parsed with the relvant frontmatter. ```python # articles/view.py class ArticleMetaView(View): model = Article def post(self, request, *args, **kwargs): data = json.loads(json.dumps(dict(request.POST))) loaded_frontmatter = frontmatter.loads(data['content'][0]) # frontmatter has keys i.e. attributes like title, description, etc. if dict(loaded_frontmatter): article_title = loaded_frontmatter['title'] article_description = loaded_frontmatter['description'] form = ArticleForm(initial={'title': article_title, 'description': article_description, 'content': loaded_frontmatter.content}) context = {'form': form} article_list = Article.objects.filter(title=article_title) if article_list: article = article_list.last() context['article'] = article return render(request, 'articles/edit_article.html', context) return render(request, 'articles/article_form.html', context) article_list = Article.objects.filter(title=data['title'][0]) # if the article title has been already taken i.e. we are updating an article if article_list: article = article_list.last() form = ArticleForm(data=request.POST) context = {'form': form} context['article'] = article return render(request, 'articles/edit_article.html', context) form = ArticleForm(data=request.POST) context = {'form': form} return render(request, 'articles/article_form.html', context) ``` In the above `ArticleMetaView` we have created a `post` method as we want to get hold of the content from the form. So, we start by extracting and converting the `request.body` data into an appropriate type for easily working with python. So, the `request.body` will contain the data like `csrf_token`, `form_data`, etc. received from the frontend template. We store the received data as `data` and now from this data, we can load the content field which will have the content information. Firstly we will extract the `request.body` which will contain the data from the form as we have made a `POST` request to this endpoint. For doing that we need to parse the content in a apropriate format such that it is python friendly. So we wrap the `request.body` into json format and then decode it back into the json string. This will give us the dict of the request data. ```python data = json.loads(json.dumps(dict(request.POST))) ``` ``` {'csrfmiddlewaretoken': ['bSYJxD39XH509tD1tZGd0WU21PUaKaLeqjjGbyzRvLXF4P8iIxb5l0fmTWVFjELQ'], 'title': ['test2'], 'description': ['test'], 'content': ['test something'], 'status': ['DRAFT'], 'blog': ['']} ``` So, this will grab the request data as a dict, we can then extract the data from this as it has data from the Form fields. We are interested in the content field in the Form, so we can get it by specifying the key `content` from the extracted data. But as we can see the data doesn't contain the actual data instead it is wrapped in a list i.e. `['test something']`, so we will have to index it and then fetch it. ```python content_string = data['content'][0] ``` This will give us the exact content field as a string. So, we can now move into extracting the frontmatter from the fields. Now, we can use the [frontmatter](https://python-frontmatter.readthedocs.io/en/latest/index.html) library to parse the content into the [loads](https://python-frontmatter.readthedocs.io/en/latest/api.html#frontmatter.loads) funciton and extract the frontmatter if it is present in the content field. The frontmatter library has a `loads` function which takes in a string and can give out a [frontmatter.Post](https://python-frontmatter.readthedocs.io/en/latest/api.html#post-objects) object. The loads function is differnet from the [load](https://python-frontmatter.readthedocs.io/en/latest/api.html#frontmatter.load) function as the load frunciton is for reading data from a stream of bytes i.e. a file or othe related byte object. The differnece is subtle but it took a read at the [documentation](https://python-frontmatter.readthedocs.io/en/latest/api.html#module-frontmatter). ```python post = data['content'][0] loaded_frontmatter = frontmatter.loads(post) ``` This wil load the content and give us a `frontmatter.Post` as said earlier. This will contain a dict with all the frontmatter if it has any and will by default parse the non-frontmatter data i.e. the remaining text into the `content` key. We need a chack if the Form field had any fronmatter this can be checked by the `dict(loaded_frontmatter)` which will return None if it cannot load the frontmatter. ```python loaded_frontmatter = frontmatter.loads(data['content'][0]) if dict(loaded_frontmatter): print(loaded_frontmatter.keys()) ``` ``` dict_keys(['templateKey', 'title', 'description', 'date', 'status']) ``` So once we have the frontmatter loaded we can get specific keys from it and initialize the form vaues to them. But we have made clear distictions that we want to perform a specific task if we have frontmatter keys in the content field of the Form else we can do something else. First let's handle the loading of the frontmatter into the form. For doing that we will get all the required attributes from the frontmatter like `title`, `description`, `content`, etc which can be accessed normally as we extract the value from a key in a dict. Once we have got those keys, we can start filling in the Form data with initial values. The [Django Model form](https://docs.djangoproject.com/en/4.0/topics/forms/modelforms/) takes in a parameter like [initial](https://docs.djangoproject.com/en/4.0/topics/forms/modelforms/#providing-initial-values) which can be a dict of the fiields along with the value that can be used for rendering the form initially when we load the template. ```python article_title = loaded_frontmatter['title'] article_description = loaded_frontmatter['description'] form = ArticleForm(initial={'title': article_title, 'description': article_description, 'content': loaded_frontmatter.content}) ``` This will take in a `ArticleForm` and fill the initial values like `title`, `description`, etc which we have provided in the dict with the values. Now, we need to parse this form in the current template or re-render the template. But before that, we need to parse this context into the template. We will create a dict with `form` as the key which can be used to render in the template. Also, we have a two ways here, either the user is creating a new article or it is updating a existing article. We need to make sure that we preserve the initial fields in the form as we are updating the existing article. So, we can filter the article objects as per the title of the current title and then if we find a article with that title, we will parse the context with that article object. ```python article_list = Article.objects.filter(title=article_title) if article_list: article = article_list.last() context{ 'form': form, 'article': article } return render(request, 'articles/edit_article.html', context) context = {'form': form} return render(request, 'articles/article_form.html', context) ``` Now, we have form data along with the article instance used for rendering the form with appropriate content. So, this will work for editing an already existing article. For a new article, we have to simply parse the form to the template and it will render the title picked from the fotnmatter or leave it empty. Similarly, for the article with no frontmatter we will iterate over the article and if the article's title already exist, we will render the article data with the form else render the form with the parsed title and other meta-data in the form. So that is how we render the form data with frontmatter into appropriate meta-data in the form. We have used Django forms and make use of HTMX for the dynamic updation of form.

Map Vimscript Keymaps to Lua with a single function

Mon, 11 Jul 2022 00:00:00 UTC

## Introduction Are you bored of writing all the keymaps from vimscript to lua? Try the below function to create all your keymaps to lua equivalent maps in Neovim. Take your vimscript keymaps and put them in lua don't write any lua for it ;) ## The Lua Function The below-provided snippet is a lua function that takes in a table of strings(list of strings), the strings will be your keymaps. The function then maps these keymaps using lua functions. You don't have to type out all the keymaps by yourself. It can also print out the lua equivalent function calls required to map the existing keymaps from vimscript to lua runtime in Neovim. Though it won't handle all the options, we have passed in a default value to the keymap. ```lua function key_mapper(keymaps) for _, keymap in ipairs(keymaps) do local mode = keymap:sub(1,1) local delimiter = " " local lhs = '' local rhs_parts = {} local m = 0 local options = {noremap = true} for matches in (keymap..delimiter):gmatch("(.-)"..delimiter) do if m == 1 then lhs = matches end if m >= 2 then table.insert(rhs_parts, matches) end m = m + 1 end rhs = '' for _, p in ipairs(rhs_parts) do rhs = rhs .. " " .. p end --print("vim.keymap.set(".."\'"..mode.."\'"..", ".."\'"..lhs.."\'"..", ".."\'"..rhs.."\'"..", "..vim.inspect(options)..")") vim.keymap.set(mode, lhs, rhs, options) end end ``` You can uncomment the print statement **once** to grab the keymaps and paste them into the config file. If you leave it uncommented, it might print every time you open up a new neovim instance. The function can be called like below: ```lua key_mapper({ 'nnoremap cpp :!c++ % -o %:r && %:ri', 'nnoremap c, :!gcc % -o %:r && %:r', 'nnoremap py :!python %', 'nnoremap go :!go run %', 'nnoremap sh :!bash %' }) ``` ![Keymapper demonstration](https://res.cloudinary.com/techstructive-blog/image/upload/v1657559501/blog-media/neovim-lua-keymapper.gif) We pass in a table of strings, these strings are just the vimscript keymaps. This function call will then map the keymaps into equivalent lua maps. You can customize it as per your needs. For further references, you can check out my [dotfiles](https://github.com/Mr-Destructive/dotfiles) on GitHub. ## How the function works The function is simply a text scrapping from lua strings. We extract the first character in the string for the mode, grab the strings which are space-separated and finally sort out which are lhs and rhs sides of the maps. We iterate over the table in lua with the help of ipairs function which allows us to iterate over an ordered list of items in a table. Using the gmatch function, we find a pattern to split the string with the space as the delimiter. Thereby, we can have separate sets of strings identified as rhs and lhs. We can store them in variables as strings as the lua functions require them as strings. We simply add those variables into the [vim.keymap.set](https://neovim.io/doc/user/lua.html#:~:text=set(%7Bmode%7D%2C%20%7Blhs%7D%2C%20%7Brhs%7D%2C%20%7Bopts%7D)%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20*vim.keymap.set()*) or [vim.api.nvim_set_keymap](https://neovim.io/doc/user/api.html#nvim_set_keymap():~:text=nvim_set_keymap(%7Bmode%7D%2C%20%7Blhs%7D%2C%20%7Brhs%7D%2C%20%7B*opts%7D)%20%20%20%20%20%20%20%20%20%20%20%20%20*nvim_set_keymap()*) functions. We by default set the value of `{noremap: True}` to avoid teh recursive mapping of the keys. These option parameter is the one which needs to be a bit more dynamic in terms of wide variety of keymaps. So, this is how we can convert the vimscript keymaps to lua in Neovim. Hope you found this useful. Thanks for reading. Happy Viming :)

Django App from Scratch Using Docker with Debian Base Image

Tue, 24 May 2022 00:00:00 UTC

## Pull a Fresh Debian Image Create a docker container from a Debian image, the following command can be used to pull a debain 11-slim image and create a container of it, also enter into the container in a interactive environment `-it` mode. ``` docker run -v $(pwd):/var/www --rm -it -p 8001:80 debian:11-slim ``` ## Create a Django App from Shell script Now, since we are inside a Debian container, we can enter a few commands, you can refer to the Mark Gibney's [GitHub repository](https://github.com/no-gravity/web_app_from_scratch) for the script. ``` apt update apt install wget wget https://raw.githubusercontent.com/no-gravity/web_app_from_scratch/main/django.sh ``` Also, if you want to do a few adjustments, you can install an editor, get used to vim or use nano. ``` apt install vim OR apt install nano ``` ``` chmod +x django.sh bash django.sh ``` I also have a few adjustment of the original script, that accepts a project name and creates a django project based on the positional parameter given to it. You can get it from the [quick-setup-script repository](https://github.com/Mr-Destructive/quick-setup-scripts/blob/main/django_docker.sh) or directly the [script](https://raw.githubusercontent.com/Mr-Destructive/quick-setup-scripts/main/django_docker.sh). To use the above file, you need to execute the command as : ``` chmod +x django_docker.sh bash django_docker.sh ``` This will generate the project in the `/var/www/` folder with the name of the project. The script will prompt you with a few things for setting up at some iterations like basic application setup , `static file` configuration, `basic tempalte` setup and the `user authentication` setup. ## Copy the contents from the docker container You can copy the contents of the folder into your local machine by entering the [cp](https://docs.docker.com/engine/reference/commandline/cp/) command in docker. ``` docker cp :/var/www/ /path/in_local_machine/ ``` This will copy the project in the docker container into the local machine for you to extend and tweak it as per your needs. That's a basic Django Project Setup by using Docker with a Debian Image.

Autoformat Python file with Black after saving in Vim

Tue, 29 Mar 2022 00:00:00 UTC

If you are like me who writes Python very badly, it has empty lines with whitespaces, no proper format in assigning variables, not formatted according to [PEP 8](https://peps.python.org/pep-0008/) standards, and you use Vim as your text editor then my friend you need a autocmd badly for it. ## Install Black in Python Install the [black](https://pypi.org/project/black/) package in python globally or locally as per your preferences. ``` pip install black ``` OR with pipx ``` pipx install black ``` For a detailed guide about running packages with pipx head toward my article on [pipx](https://mr-destructive.github.io/techstructive-blog/pipx-intro/). ## Set up Autocmd in Vim We can set up a autocmd. What is a autocmd? It is about running commands when certain events occur like running a command when a file is saved, a buffer is opened or closed, and so on. What we want is to run the black command from the shell when the current file is saved. So, we can create a autocmd as follows: ```vimscript autocmd BufWritePost * !black % ``` Now, this will run when any type of file is saved, so we will make it specific to python by adding a `*.py` to add in the autocmd. ```vimscript autocmd BufWritePost *.py !black % ``` This works, but it gives a prompt after the command has been executed, to run the command silently we can simply add the silent keyword before the execution of the command from the shell. ```vimscript autocmd BufWritePost *.py silent !black % ``` This looks perfect! But still, we need to add a auto-group(`augroup`) that groups the autocmds and by adding `autocmd!` it will clear all the commands from the group. ```vimscript augroup python_format autocmd! autocmd BufWritePost *.py silent !black % augroup end ``` We can now add it to the vimrc to work all the time. ## Using pipx If you have used pipx to install black, you need to setup the autocmd a bit differently. ```vimscript autocmd BufWritePost *.py silent !pipx run black % ``` It might be a bit slower than running with global installation, but it is a neat way to run python package. So, that's it we can now write clean and safe python code without breaking a sweat in Vim. Happy Coding :)

Python: Search and Replace in File

Mon, 28 Mar 2022 00:00:00 UTC

## Searching and Replacing the text in a File Using simple python semantics, we can perform search and replace in a file. Firstly, we will define the file name, along with the words to search and replace. After defining the sets of variables, we will open the file in `r+` mode i.e. we can perform read as well as write operations in the file. We will store the entire file contents using the [read](https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files) function, the contents of file are now stored in the form of a string. We further can set the position of the cursor or the current position in the file using the [seek](https://python-reference.readthedocs.io/en/latest/docs/file/seek.html) function. The seek function takes in a optional parameter as the position to set for reading/writing of file. Using the [truncate](https://python-reference.readthedocs.io/en/latest/docs/file/truncate.html) function, we can clear all the contents of the file. Finally, we generate the content by replacing the words i.e. the old word with the new word from the string which we store the contents. After replacing the content in the string, we write the string variable into the file and hence the substitution was performed in the text file. ```python file_name = 'temp.txt' old_text = 'foo' new_text = 'python' with open(file_name, "r+") as fname: lines = fname.read() fname.seek(0) fname.truncate(0) subs = lines.replace(old_text, new_text) fname.write(subs) ``` ![Python Search Replace in File](https://res.cloudinary.com/techstructive-blog/image/upload/v1648479344/blog-media/cstvfdlazyfriwvnilju.png)

Vim: Get the Text from Visual Selection

Sat, 05 Mar 2022 00:00:00 UTC

## Using Registers We can get the selected text in a variable in Vim Script using registers. ```vimscript normal gv"xy let context = getreg("x") ``` Lets break down the command ``` normal mode -> gv -> (y)ank text -> to the "x" register | | | Copy the contents into x register(or any register you like) | Select the previously selected text ``` Here, we are entering normal mode and selecting text which was previously selected and yank the contents into a register in this case we are using (x) register, it can be any register. Now to get the contents of that register we can use the function `getreg("register_name")` or use `"xp"` to paste the contents of the `x` register or more generally for any register(`"p`). Hence we can store the contents of the selected text in a variable for further processing or manipulation. To quickly test this snippet from command mode, you can try the following steps: Select a text and press Escape, we just want the `gv` command to refresh and get it's contents to the latest visual selection. ```vimscript :normal! gv"xy ``` ```vimscript :let foo = getreg("x") ``` ```vimscript :echo foo ``` The echo command will simply print the text which we have selected in the file. ![Visual Select Text](https://res.cloudinary.com/techstructive-blog/image/upload/v1646483173/blog-media/wlrxgtmegtycilyhvyiz.gif)

Dockerize a Django project

Wed, 02 Mar 2022 00:00:00 UTC

## Dockerize a Django project We can run our Django projects in a Docker Container by creating a Docker image for our project. It is really easy and intuitive to create a Dockerfile for any given application as it really is a matter of writing commands in a file and basically running it is a isolated environment. To create a Docker image, we first need a Dockerfile. A Dockerfile is simply a Blueprint to create a image in Docker. In this file we specify the instructions/commands/environment variables to create a image for our app to run. ### Create a Docker File To create a dockerfile, simply create a file named as `Dockerfile` without any extension(not a .txt file). We need to then add the following code into the file. ```dockerfile FROM python:3.9-buster ENV PYTHONUNBUFFERED=1 WORKDIR /code COPY requirements.txt . RUN pip install -r requirements.txt COPY . . EXPOSE 8000 CMD ["python","manage.py","runserver","0.0.0.0:8000"] ``` - Let's see what are we doing here, first we are pulling a base image of Python in this case it is [3.9-buster](https://github.com/docker-library/python/blob/a4b368154b7e3c33c76385f1be7a998fcf3123eb/3.9/buster/Dockerfile), but it can be any of the [Official Python images](https://hub.docker.com/_/python) from [dockerhub](https://hub.docker.com). It is simply a environment for our Django project to run. You can even use linux environments like ubuntu, debian or alpine. - Next we add a `PYTHONUNBUFFERED` as a environment variable and initialize it to `1`, which basically allows to parse python output straight into the terminal without actually buffering first. - We set the `WORKDIR` as the `code` directory, this sets the provided directory as the base to run any command or instructions in the Dockerfile. - We then `COPY` the `requirements.txt` file in the current(`.`) directory which is `code` as mentioned earlier. We simply want the `requirements.txt` file in the current directory so we can proceed with the next step which is to install the dependencies for the Django project - After copying the `requirements.txt` file, we simply install the packages mentioned int the file with `pip` we use the `RUN` command to execute any shell commands in the environment. - Next step is to `COPY` all the files from the current folder(local machine) into the docker environment image. So, we have the source code files in the Container. - Expose the port `8000`(default) or any other port you would want to run Django in the image. We use the expose command in complement to the `-p` argument when we want to create the container. I have explained the [port-forwarding](https://mr-destructive.github.io/techstructive-blog/docker-port-forward) concept in the previous TIL. - Finally we can run the server. We need to parse every command as a comma separated string in the list to the `CMD` which the container actually executes the command when the image is build. So, all the commands previously were to build an image for the django project, the `CMD` will actually run the server in a container after building the image of the django project. ### Build the image Using the Dockerfile, we can create the image for the Django project by a simple command. ``` docker build . -t django-app ``` Assuming the Dockerfile is in the same folder from which you are running this command. ![Build the django-app image](https://res.cloudinary.com/techstructive-blog/image/upload/v1646230907/blog-media/jj04subyvkuvfb5obytu.png) ![Build Complete Success](https://res.cloudinary.com/techstructive-blog/image/upload/v1646230988/blog-media/ugjoakqgyhiwelqkyaat.png) ### Run the Image and Create a Container ``` docker run -p 8000:8000 django-app ``` ![Create Container](https://res.cloudinary.com/techstructive-blog/image/upload/v1646231023/blog-media/yneuz46burorz4b5vzp4.png) You can use any port like `3000` or `4000` on your local system by changing the first port number as `3000:8000` and so on. So, the first port number is the local system's port number and the port number followed by a `:` is the port number for the environment inside the container of the django project. The container port is the same which we have `exposed` while creating the image. You can now go to [127.0.0.1:8000](http://127.0.0.1:8000) or your chosen port in your local environment and see the django app running. If you are using a Docker as a Virtual Machine, you need to find the IP-address of that Virtual-Machine(`docker-machine ip`) and use that instead of `localhost` or `127.0.0.1`. ### Cleaning the Container for iterations We need to stop the container to run it again after making a few changes in the app if any. Simply pressing the `CTRL + D` option wont stop the container here. We need to stop the container by executing: ``` docker ps docker stop ``` This will stop the container. Also if you want to remove the images you built, you can run the command: ``` docker image ls docker rmi -f ``` So, we now have a image of our Django project, this image can be used by anyone inside a docker environment and thus creating much more easier to test/work on a given project irrespective of the system is being used.

Docker Port Forwarding

Tue, 01 Mar 2022 00:00:00 UTC

## Docker Port Forwarding Port forwarding is a process to redirect the communication of one address to other. It is also known as Port Binding. We can use `-p` command to use port forwarding in our local Docker environment. ``` docker run -p 8000:8000 django-app ``` The first port number is the local machine port and followed by a `:` is the container port number. SO, the request from the container port are forwarded to the local/outside world in the docker environment. Additionally, we need to expose the container port first. We can do that in the Dockerfile or by adding a `-e` argument followed by the port to expose. This will open the port on container to forward the requests to the specified port in the `-p` option. In the Dockerfile, we can expose the port by adding the command `EXPOSE 8000`, or any other port number.