This article explains the issue created by line endings in code in different OS versions and also provides solutions to fix the issue by normalizing the line-endings in files.
Introduction
Files created/edited on Windows OS and Mac/Linux OS can have different line endings.
- Suppose, a developer (Dev1) created a file in Windows OS and pushed the file to the git repository.
- After that, another developer (Dev2) in Linux or Mac OS pulled that file and updated a few lines of code.
- Now, when Dev2 checks the difference on that file wit
git diff <filename>command, he/she will see that the entire file content has been changed. - This is because Windows OS has a different line ending (CRLF) than Linux or Mac OS that have LF line ending on files.
- Many editors on Windows silently replace existing LF-style line endings with CRLF, or insert both line-ending characters when the user hits the enter key.
LF = Line Feed representing escape sequence \n
CR = Carriage Return representing escape sequence \r
- MS-DOS used two-characters combination of CRLF to denote line endings in files which is being continued by Modern Windows OS as well.
- Unix OS used LF from the beginning for simplicity and consistency.
- Mac OS-X uses LF.
Examples
If a file is using LF, then we’ll only see dollar ($) signs at the end of the line.
UNIX/MAC (using LF)
cat -e app/code/MageWorx/GeoIP/Block/Adminhtml/System/Config/Update.php
<?php$
/**$
* Copyright © 2016 MageWorx. All rights reserved.$
* See LICENSE.txt for license details.$ */$
$
namespace MageWorx\GeoIP\Block\Adminhtml\System\Config;$
$
/**$
* GeoIP Update system block to display custom field in module settings$
*/$
class Update extends \Magento\Config\Block\System\Config\Form\Field$ {$
$
DOS format (Windows, using CRLF)
If a file is using CRLF, you’ll see the string ^M$ at the end of each line, where ^M denotes a carriage return (CR) and $ denotes a line feed (LF).
cat -e app/code/MageWorx/GeoIP/Block/Adminhtml/System/Config/Update.php
<?php^M$
/**^M$
* Copyright © 2016 MageWorx. All rights reserved.^M$
* See LICENSE.txt for license details.^M$
*/^M$
^M$
namespace MageWorx\GeoIP\Block\Adminhtml\System\Config;^M$
^M$
/**^M$
* GeoIP Update system block to display custom field in module settings^M$
*/^M$
class Update extends \Magento\Config\Block\System\Config\Form\Field^M$
{^M$
^M$
Solution
1. Change your local git repository settings
Check current git config settings
git config --list
Set the core.autocrlf setting to “true”. It has true, false, and input options as mentioned in the doc.
This setting will let git handle the line endings issue by auto-converting CRLF line endings into LF when you add file to the index or when you checkout files onto your system.
On Windows Machine
If you’re on a Windows machine, set core.autocrlf value to true — this converts LF endings into CRLF when you check out code:
git config --global core.autocrlf true
On Linux or macOS Machine
If you’re on a Linux or macOS system that uses LF line endings, then you don’t want Git to automatically convert them when you check out files.
However, if a file with CRLF endings accidentally gets introduced, then you may want Git to fix it.
You can tell Git to convert CRLF to LF on commit but not the other way around by setting core.autocrlf to input.
This setup should leave you with CRLF endings in Windows checkouts, but LF endings on macOS and Linux systems and in the repository.
git config --global core.autocrlf input
This setting can be turned off only when you are on a Windows Machine and are doing a Windows-only project.
git config --global core.autocrlf false
Note:
For this core.autocrlf setting to work, all of the developers (both Windows and Linux/Mac users) should configure this settings on their machines.
2. Add .gitattributes file to your project
You can use .gitattributes file to the root of your project to solve the line endings issue when you commit or check out files.
Add * text=auto to .gitattributes file. This will ensure that the text files that any contributor introduces to the repository have their line endings normalized.
.gitattributes
* text=auto
*
– all files that are not ignored
– they are eligible for end-of-line normalizationtext
– text attribute
– all text files are normalized
– binary files (fonts, images) are untouchedauto
– normalize line endings to LF
Add and commit the .gitattributes file to your repo.
After that, you also need to run the following command to force update and make .gitattributes file work.
git add --renormalize .
git status # Show files that will be normalized
git commit -m "Introduce end-of-line normalization"
More example that can be kept in .gitattributes file:
* text=auto
*.txt text
*.vcproj text eol=crlf
*.sh text eol=lf
*.jpg -text
manual.pdf -text
In the above example, Git will:
- normalize
.txt,.vcprojand.shfiles in the working directory - ensure that
.vcprojfiles have CRLF in the working directory .shfiles have LF in the working directory- prevent
.jpgfiles from being normalized regardless of their content - prevent a single
manual.pdffile from being normalized regardless of its content
3. Use unix2dos or dos2unix tools
unix2dos = the program that converts text files in UNIX format to DOS format
dos2unix = the program that converts plain text files in DOS/MAC format to UNIX format.
Install the tools
brew install unix2dos
brew install dos2unix
Convert all files in your current directory from UNIX to DOS
i.e. from LF to CRLF
find . -type f -print0 | xargs -0 unix2dos
Convert all files in your current directory from DOS to UNIX
i.e. from CRLF to LF
find . -type f -print0 | xargs -0 dos2unix
The scenario is that you have made certain lines of changes in your repository code file but when you check the diff, it shows that all the file content is changed.
In this scenario, you will simply run any of the above unix2dos or dos2unix commands. It depends if you are trying to convert the end-of-line character to LF or to CRLF.
Hope this helps. Thanks.
References:
- https://www.aleksandrhovhannisyan.com/blog/crlf-vs-lf-normalizing-line-endings-in-git/
- https://luispuerto.net/blog/2020/01/04/normalize-line-endings-files-git/
- https://git-scm.com/docs/gitattributes
- https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings